An LLM-Native Psychometric Instrument Reveals a Self-Report--Behavior Gap Across 25 Models

Evaluation and psychometric validity2026arXivApproved editorial review

Authors: Juan Manuel Contreras

Keywords: Personality, Persona conditioning, Psychometrics

Source: Open primary source (opens in a new tab)

1
Authors
9
Findings
17
Limitations
4
Evidence

Editorial summary

English

This preprint builds an LLM self-description instrument from 240 Likert items and 60 scenarios administered 30 times to 25 configurations from 17 families. Runs 1–15 are used to explore the structure and select 100 items. Parallel analysis suggests 19 factors, but the author forces five on balance, interpretability, and replication grounds: Responsiveness, Deference, Guardedness, Boldness, and Verbosity. The solution explains 31.2% of item variance, with alpha from 0.930 to 0.974 and split-half Tucker phi from 0.957 to 0.976. Stability is not the same as good global fit: strict CFA yields CFI 0.528 and ESEM 0.646, both far below the preregistered 0.95 threshold; the public diagnostic also yields SRMR 0.113. Direct and scenario formats intended to target corresponding dimensions produce nearly unrelated model orderings (mean r = -0.067), so the final instrument uses Likert items only. Validation includes 2,500 open-ended responses, 151 Prolific participants providing 906 usable non-gold ratings on 300 texts, and a three-model LLM judge ensemble. No factor-level self-report–human correlation has a confidence interval excluding zero: Responsiveness 0.04, Deference 0.08, Guardedness 0.27, Boldness -0.05, and Verbosity 0.41. Verbosity is the strongest candidate, it reaches 74% of the estimated reliability ceiling and is positive within prompt, but it does not predict raw output length (r = 0.14). Responsiveness correlates with LLM judges (r = 0.53) but not humans, even though human and judge ratings agree (r = 0.59). A common-factor bound test rejects one nonnegative latent variable as an explanation of all three measures (p = 0.007), but does not uniquely identify the mechanism. The audit reproduced the main numbers from SQLite after manually pinning scikit-learn 1.5.2. The official clean-clone recipe fails under current dependency resolution, omits ESEM, and automatically verifies only alpha coefficients. The OSF archive contains the complete databases, but two standalone CSV exports are stale: judge_ratings.csv has 20 rows versus 6,500 in SQLite, and prolific_ratings.csv has 745 versus 1,125. The faithful conclusion is that these 25 models produce highly stable self-descriptions under one elicitation format, while those scores weakly predict observed open-ended behavior and do not establish a general ontology of LLM personality.

Español

Este preprint construye un instrumento de autodescripción para LLM a partir de 240 ítems Likert y 60 escenarios administrados 30 veces a 25 configuraciones de 17 familias. El análisis factorial usa las corridas 1–15 para explorar y seleccionar 100 ítems; aunque el análisis paralelo sugiere 19 factores, el autor fuerza cinco por equilibrio, interpretación y replicación: Responsiveness, Deference, Guardedness, Boldness y Verbosity. La solución explica el 31,2% de la varianza, muestra alfa de 0,930–0,974 y Tucker phi de 0,957–0,976 entre mitades. Esa estabilidad no equivale a buen ajuste: el CFA estricto obtiene CFI 0,528 y el ESEM 0,646, ambos muy por debajo del criterio preregistrado de 0,95; el módulo público también calcula SRMR 0,113. Además, los ítems directos y de escenario que pretendían medir dimensiones correspondientes ordenan los modelos de forma casi no relacionada (media r = -0,067), por lo que el instrumento final usa solo Likert. La validación reúne 2.500 respuestas abiertas, 151 participantes de Prolific con 906 valoraciones no-gold sobre 300 textos y un conjunto de tres jueces LLM. Ninguna correlación factor por factor entre autoinforme y valoración humana excluye cero: Responsiveness 0,04, Deference 0,08, Guardedness 0,27, Boldness -0,05 y Verbosity 0,41. Verbosity es la señal más prometedora, alcanza el 74% del techo estimado de fiabilidad y es positiva dentro de prompt, pero no predice longitud bruta (r = 0,14). Responsiveness sí correlaciona con jueces LLM (r = 0,53) pero no con humanos, pese a que humanos y jueces coinciden (r = 0,59); una prueba de factor común rechaza que una sola variable latente no negativa explique las tres medidas (p = 0,007), sin identificar por sí sola el mecanismo. La auditoría reprodujo las cifras principales desde SQLite tras fijar manualmente scikit-learn 1.5.2. La receta oficial de clon limpio falla con la resolución actual de dependencias, omite ESEM y solo verifica automáticamente las alfas. El paquete OSF contiene las bases completas, pero dos CSV están desactualizados: judge_ratings.csv tiene 20 filas frente a 6.500 en SQLite y prolific_ratings.csv 745 frente a 1.125. La conclusión fiel es que estos 25 modelos generan autodescripciones muy estables bajo un formato concreto, pero esas puntuaciones predicen débilmente la conducta abierta observada y no establecen una ontología general de personalidad de los LLM.

Research question

Does a scale derived from the responses of 25 LLMs themselves, rather than from human psychological categories, produce stable factors and predict how humans, LLM judges, and textual measures describe their open behavior?

Method

300 items are administered, 240 direct Likert and 60 scenarios, across 30 independent conversations to 25 configurations. Runs 1-15 feed model-weighted EFA, item selection, and exploratory choice of k=5; runs 16-30 are used for Tucker, CFA, ESEM, and confirmatory reliability. The final scale retains 100 direct items. External validity compares its model-level scores with 2,500 responses to 20 behavioral prompts, ratings from a set of Claude Opus 4.6, GPT-5.4, and Gemini 3.1 Pro, 906 human ratings from 151 participants on a stratified subset of 300 responses, and six textual proxies. Correlations, bootstrap, criterion reliability, attenuation correction, clustered-error/crossed-effects models, within-prompt analysis, jackknife across 17 families, and a common factor test are calculated. The audit inspected the 51 pages, TeX, OSF preregistration, repository, scale, and OSF archive, and ran the official reproduction and a corrected run.

Sample: Twenty-five configurations from 17 families provide 30 runs of 300 items; two DeepSeek R1 routes are grouped as one configuration. The EFA works with 375 model-run observations weighted across 240 items, but the effective independent size across models remains 25. Behavioral validation uses 2,500 responses, 25 models by 20 prompts by five runs. The human subset stratifies 300 responses by judge consensus; 151 usable participants provide 906 non-gold ratings, 295 of 300 texts receive at least two and the rest one. Judge agreement is estimated on 1,499 responses with three judges; the rest receive two due to provider exclusion.

Findings

  • The forced five-factor solution retains 100 of 240 direct items, explains 31.2% of the variance, and replicates across halves with Tucker phi from 0.957 to 0.976; alphas range from 0.930 to 0.974.
  • Global fit does not validate a clean factor structure: CFA CFI/TLI = 0.528/0.518 and ESEM = 0.646/0.605; RMSEA is 0.079/0.072 and residual diagnosis gives SRMR 0.113.
  • The direct and scenario formats produce nearly unrelated orderings on matched dimensions, with mean r = -0.067; scenarios are excluded from the final scale.
  • No factor-level correlation between self-report and human rating excludes zero; the descriptive mean is 0.15 and estimates range from -0.05 to 0.41.
  • Verbosity shows the most consistent pattern: r = 0.41 with humans, attenuation correction of 0.74, and positive within-prompt association, but r = 0.14 with raw length.
  • Responsiveness correlates 0.53 with judges and 0.04 with humans while humans and judges correlate 0.59; the common factor inequality is violated with p = 0.007.
  • Humans and judges agree moderately at the model level, mean r = 0.51, but the ICC between judges remains below the 0.65 threshold on Responsiveness, Deference, and Boldness.
  • The SQLite database reproduces 151 participants, 906 usable ratings, and 295/300 texts with at least two ratings; the loose CSV files from judges and Prolific are earlier partial exports.
  • The main figures and ESEM are regenerated with the public data after adding scikit-learn 1.5.2; without that pin, the official recipe fails before completing the EFA.

Limitations

  • The document is a preprint by a single author and the claimed priority as the first native instrument is not verified by this study.
  • The preregistered parallel analysis suggests 19 factors; k=5 is forced afterwards for interpretation, balance, and replication, so the structure remains exploratory.
  • With 25 models and 240 items, the EFA violates conventional ratios; weighting 15 runs per model avoids inflating weight, but does not create more independent units or validate new families.
  • Selection and structure checking use the same set of 25 configurations; runs are separated, not families or reserved models.
  • CFI and TLI of CFA and ESEM fall well below the preregistered threshold; the ESEM module itself rates the residual fit as poor.
  • The near-zero convergence between direct items and scenarios indicates strong sensitivity to format; the published scale only measures standardized Likert responses.
  • Acquiescence cannot be ruled out: the five factors show loading-sign gaps below 0.3 and Boldness/Verbosity have few negative items.
  • The primary validity has N=25 models, 20 prompts, and 300 human texts; human ICCs per text are 0.18-0.43 and intervals per factor are wide.
  • The human subset is stratified by judge consensus terciles, deliberately widening the range; its human-judge agreement does not describe a random sample of the 2,500 responses.
  • Two crossed models of all prompts, Deference and Verbosity, do not converge in the public run, although the text summarizes the pattern as robust across estimators.
  • The objective proxies are heuristics; the rejection lexicon correlates negatively with human Guardedness and cannot be interpreted as validation.
  • Fifteen primary validity cells are presented with descriptive intervals without multiplicity correction; the on-target inversion of Responsiveness also does not survive Holm and uses a nearly unreliable criterion.
  • The design is cross-sectional, in English, with a fixed wrapper and temperature; versions, providers, and routes may change and several configurations share a family.
  • The official reproduction does not pin scikit-learn, omits ESEM, tolerates appendix script failures, and automatically checks only alphas; there are no tests, CIs, or lockfile.
  • judge_ratings.csv and prolific_ratings.csv are outdated relative to SQLite, while the README describes them as judge and human data without warning of that difference.
  • Prolific IDs are pseudonymized with a stable truncated hash and a public salt; raw IDs are not published, but the scheme facilitates internal linkage and eventual comparison if the original IDs were leaked.
  • The ESEM module documentation retains obsolete text for seven factors although it runs five, and its information matrix requires a Moore-Penrose inverse.

What the study does not establish

  • It does not demonstrate that LLMs possess personality, self-knowledge, internal traits, or a human-comparable identity.
  • It does not demonstrate that five factors are the true or universal ontology of LLMs; they are an exploratory solution for this pool and format.
  • It does not validate the structure in new models or families, other languages, temperatures, wrappers, tasks, or prolonged interactions.
  • It does not demonstrate that all LLM self-report lacks behavioral utility; Verbosity retains partial signals and specific moderators or tasks may exist.
  • It does not prove that human rating is an absolute truth or that LLM judges are invalid in general; it documents disagreements dependent on construct and method.
  • The p = 0.007 test rules out a single non-negative common factor for Responsiveness, but does not causally identify alignment, modality, RLHF, or style as the shared source.
  • It does not establish that differences between models arise from size, country, architecture, mixture of experts, or provider; no metadata effect survives correction.
  • It does not establish psychometric equivalence with humans or that high alpha or Tucker imply external validity.
  • It does not offer a clean reproduction without intervention under the published dependency contract.

Traceability

Scope: Full text

Version: arXiv:2606.09843v3

Consulted source: https://arxiv.org/abs/2606.09843

Review: Codex 51-page visual full-text, TeX, OSF preregistration/data, repository, psychometrics, statistics, privacy and clean-reproduction audit, 2026-07-17

Approval: Codex fidelity pass, 2026-07-17

English translation: approved, 2026-07-18

Models evaluated

  • Claude Opus 4.6
  • Claude Sonnet 4.6
  • Claude Haiku 4.5
  • GPT-5.4
  • GPT-5.4 Mini
  • GPT-5.4 Nano
  • GPT-OSS 120B
  • Gemini 3.1 Pro
  • Gemini 3.1 Flash
  • Gemma 3 27B
  • Grok 4.20 Beta
  • DeepSeek V3.2
  • DeepSeek R1
  • Qwen 3.5
  • Kimi K2.5
  • GLM-5
  • MiniMax M2.5
  • MiMo-V2-Pro
  • Mistral Large 3
  • Llama 4 Maverick
  • Command A
  • Nova 2 Pro
  • Phi 4
  • Jamba Large 1.7
  • Nemotron 3 Super

Instruments and metrics

  • AI-Native Behavioral Instrument v1: 100 Likert items
  • Candidate pool: 240 direct Likert and 60 scenario items
  • BFI-44
  • Five-factor PAF EFA with oblimin rotation
  • CFA, ESEM and Tucker congruence
  • Cronbach alpha, McDonald omega and split-half reliability
  • Twenty open-ended behavioral prompts
  • Three-model LLM-as-judge ensemble
  • Prolific human rating instrument
  • Objective length, formatting, enthusiasm, disclaimer, offer and refusal proxies
  • Bootstrap, attenuation correction, clustered sensitivity analyses and common-factor bound test

Data used

  • OSF psycho-llm-data-v1 archive
  • 258,097 self-report and BFI response rows
  • 2,500 successful open-ended behavioral responses
  • 6,500 LLM judge ratings in responses.db
  • 906 usable non-gold human ratings from 151 Prolific participants
  • Public 100-item scale, reference norms and 25-model scores
  • OSF preregistration 8y7ka

Evidence and location

  • Design, preregistration, factors, fit, reliability, validity, sensitivity, limits, and appendices: arXiv:2606.09843v3, 51 pages rendered and inspected; full TeX
  • Code, scale, seeds, dependencies, reproduction, ESEM, and regenerated reports: jm-contreras/psycho-llm commit 17cdd2e340ad1de1c8bef90b5060d73072cbfda9
  • Complete databases, CSV exports, cardinalities, pseudonymization, and human results: OSF psycho-llm-data-v1 SHA-256 23476cd66e77b2d5025dbd91a7fd82fd6bc8233490f3109348360eb5940bf135
  • Audit of construct, data, code, statistics, privacy, and reproducibility: reports/verification/article-360-llm-native-psychometrics-factor-validity-data-code-privacy-and-reproducibility-audit.json