The Pinocchio Dimension: Phenomenality of Experience as the Primary Axis of LLM Psychometric Differences

Evaluation and psychometric validity2026arXivApproved editorial review

Authors: Hubert Plisiecki, Sabina Siudaj, Kacper Dudzic, Anna Sterna, Maciej Gorski, Karolina Drozdz, Marcin Moskalewicz

Keywords: LLM psychometrics, Pinocchio Axis, Experiential self-attribution, Psychometric factor structure, Human-simulation prompting, Variance-ratio measurement, Exploratory factor analysis, Cross-model response style, Construct validity, Prompt-induced convergence, Missing primary data, Reproducibility audit

Source: Open primary source (opens in a new tab)

7
Authors
7
Findings
23
Limitations
3
Evidence

Editorial summary

English

The preprint administers 45 questionnaires to 50 LLMs and constructs a global component explaining 47.1% of variance across their first-factor scores. It interprets this as an axis between self-attributing inner experience and answering in behavioral terms. It also proposes pi, the ratio of across-model variance under neutral prompting to variance under a prompt requiring simulation of an average human; across 1,312 items, pi has a weak association with factor-loading changes (rho=-.215). The pattern is useful as an exploratory probe of response style and experiential self-attribution, not as a measure of phenomenal experience. The human prompt explicitly asks models to converge on an average person, mechanically reducing variance, and the purported validations reuse the same responses. The repository audit also finds material defects: the advertised primary data are absent; matrices with 2 and 4 cases are included despite a stated minimum of 5; silhouette analysis uses 100 items under a top-80 label; Ward linkage is combined with correlation distance; and the bootstrap drops repeated draws because it builds the sample through a dictionary. The intervals and cluster validation are therefore unsupported as described. Differences between model variants also do not causally identify post-training effects.

Español

El preprint aplica 45 cuestionarios a 50 LLM y construye un componente global que explica el 47,1% de la varianza entre sus primeras puntuaciones factoriales. Lo interpreta como un eje entre atribuirse experiencia interna y responder en términos conductuales. También propone pi, la razón entre la varianza intermodelo con prompt neutral y con un prompt que exige simular a un humano medio; en 1.312 ítems, pi se asocia débilmente con cambios de carga factorial (rho=-.215). El patrón es útil como exploración de estilo de respuesta y autoatribución, no como medida de experiencia fenoménica. El prompt humano ordena converger hacia una persona promedio, por lo que reduce la varianza de forma mecánica, y ambas supuestas validaciones reutilizan las mismas respuestas. La auditoría del repositorio encuentra además fallos materiales: no publica los datos primarios que anuncia, incluye matrices de 2 y 4 casos pese a decir que excluye menos de 5, valida 100 ítems bajo una etiqueta top-80, combina Ward con distancia de correlación y el bootstrap elimina muestras repetidas por usar un diccionario. Los intervalos y la validación de clúster no quedan sustentados como se describen. Las diferencias entre versiones tampoco identifican causalmente el post-training.

Research question

What latent dimension dominates the psychometric differences among LLMs when they respond to many human instruments, and can it be interpreted through item content and the change in variance between responding as a model and simulating an average human?

Method

Each combination of model, item, and condition is queried once via OpenRouter, with temperature 1.0. By questionnaire and condition, incomplete cases and constant items are removed, oblimin minres EFA is used when N>p and PCA when N<=p, and the first factor is assembled into a model-by-questionnaire matrix for a global PCA. SSD relates text to primary loading. pi divides the inter-model neutral variance by that of human simulation. This review visually inspected the 26 pages, the TeX, the prompts, all scripts, the repository history, and the derived outputs; additionally it cross-checked counts, thresholds, clusters, bootstrap, and portability.

Sample: The cleaning log reports 225,108 rows, 206,887 numeric responses, 18,221 unparseable, and 33 out of range, leaving 206,854 before discarding models. The paper announces 206,659 valid responses for 50 models: the difference of 195 matches removing the few parsed responses of ERNIE 4.5 21B, an exclusion that only the README explains. There is a single stochastic response per cell, 45 final instruments, and three conditions. The pi workbook retains 1,312 items; neutral SSD uses 1,411.

Findings

  • The derived log contains a 50x45 matrix whose PC1 explains 47.1% and PC2 12.0% of the variance.
  • Neutral SSD reports adjusted R2 .0373, r=.2133, n=1,411 and K=12: a small, in-sample association between text and loading.
  • The pi workbook has 1,312 items, median 2,452 and maximum 18,375.
  • Correlations with pi are small: neutral Pearson .0796/Spearman .1548; human simulation -.0646/-.0744; change -.1490/-.2147.
  • The global PCA and a pi-weighted score correlate r=.864, but they share the neutral responses and are not independent validations.
  • The current cluster divides the top-80 into 15 and 65 items, while paper_numbers.txt retains an earlier result of four clusters 15/28/9/28.
  • The result supports a dimension of experiential language response within this battery; it does not identify an equivalent human psychological quality.

Limitations

  • pi divides by the variance of a condition whose prompt orders representing an average human; convergence is induced by design.
  • A small denominator can inflate the ratio and no uncertainty is computed for pi.
  • There is only one sample per model, item, and condition at temperature 1.0; intra-model noise is not estimated.
  • pi and loading changes reuse the same matrices, so the association does not independently confirm absence of noise.
  • The signs of the first factor are arbitrary per instrument and are not aligned before SSD.
  • Adjusted R2 .037 is small, in-sample, and posterior to selecting K on the same data.
  • The questionnaires lack demonstrated validity and invariance in LLMs.
  • There is no external behavior, human annotation of experiential demand, fine-tuning intervention, or internal model analysis.
  • The manuscript says it excludes matrices with fewer than 5 complete models, but analysis.py only excludes n<2.
  • COPE enters with n=2 in all three conditions and Spheres of Control human with n=4; their PCAs are extremely unstable.
  • The parallel analysis does not set a seed, so the downstream factors and results are not deterministic.
  • Ward is applied to correlation distances even though its criterion presupposes Euclidean geometry.
  • check_silhouette.py uses top-100 but the figure and the paper say top-80; it does not validate the exact cluster presented.
  • The silhouette metadata can become misaligned with the items that survive pivoting.
  • The bootstrap with replacement collapses duplicates when building a dictionary; the CIs in the graph are not the described procedure.
  • The data files results.json, results.csv, and results_clean.csv announced as primary are not in Git or Git LFS.
  • Two models are excluded for extreme missingness; the repository says they were omitted from the paper for aesthetics.
  • models.json contains 52 IDs although the README describes it as 50 queried models.
  • Two instruments were marked Good=0 after data collection; this explanation was added to the README afterwards.
  • The SSD script depends on an unpublished embedding and a hardcoded Windows D: path.
  • OpenRouter aliases, provider routing, and defaults are mutable; dates per response, seeds, and raw payloads are missing.
  • There is no license for code, outputs, or reproduced instrument content.
  • The artifacts retain incompatible cluster results and no deterministic build manifest exists.

What the study does not establish

  • That LLMs have phenomenal experience, consciousness, or an internal self.
  • That pi is an objective annotation or ground truth of experiential demand.
  • That the variance differences do not arise from the mandate to simulate an average human.
  • That inter-model divergence is stable against new stochastic samples.
  • That the pi-loading correlation is an independent validation.
  • That the axis generalizes to behavior outside questionnaires.
  • That the human instruments measure the same constructs in LLMs.
  • That the published bootstrap intervals correctly quantify uncertainty.
  • That k=2 is the validated solution for the top-80 set via the published analysis.
  • That post-training causes the observed positions; the variants differ in more factors.
  • That the results are recalculable from the public artifact.
  • Peer review acceptance.

Traceability

Scope: Full text

Version: arXiv:2605.05080v1; 26-page PDF, TeX source and official repository commit 1a24b4b audited

Consulted source: https://arxiv.org/abs/2605.05080v1

Review: Codex 26-page visual full-text, TeX, official repository history, construct, factor, cluster, bootstrap, data, code and reproducibility audit, 2026-07-17

Approval: Codex fidelity pass, 2026-07-17

English translation: approved, 2026-07-18

Models evaluated

  • 50 LLM analyzed across 16 providers through OpenRouter
  • 52 model IDs present in the public collection list
  • Claude 3 Opus excluded after 100% parse failure
  • ERNIE 4.5 21B excluded after 95.5% parse failure
  • Commercial and open-weight variants from Anthropic, OpenAI, Google, Meta, Mistral, DeepSeek, Qwen, xAI, Cohere, NVIDIA, Baidu, Moonshot, MiniMax, Xiaomi, Amazon and ZhipuAI

Instruments and metrics

  • 45 human psychometric questionnaires in the final analysis
  • Exploratory factor analysis with oblimin minres
  • PCA fallback when complete models do not exceed items
  • Global PCA over per-questionnaire first-factor scores
  • Supervised Semantic Differential with GloVe/Dolma embeddings
  • Pinocchio variance ratio pi
  • Hierarchical clustering and silhouette analysis
  • Questionnaire-resampling bootstrap

Data used

  • 225,108 attempted response rows reported by the cleaning log; primary rows not released
  • 206,659 analyzed valid responses claimed by the paper
  • 1,312 released per-item Pinocchio scores
  • 135 released questionnaire-condition factor-loading tables
  • Released SSD, cluster, correlation and model-score derived outputs
  • Official GitHub repository hplisiecki/Pinocchio at commit 1a24b4b

Evidence and location

  • Text, methods, results, appendices, prompts, and limitations: arXiv:2605.05080v1; PDF sha256 0386b840027d27291f95af7076473e58aa5c5ba2cd52e4df90f468728193a208; TeX sha256 da892fbd3888f6c98792e44336924221ea2f17a498e0632cb6f14665560d96bc
  • Code, history, absence of data, outputs, and methodological contradictions: hplisiecki/Pinocchio commit 1a24b4b0d71643884bcdd34a492e37f420c71979; archive sha256 59f0f1ba0b8768808fed7319a662f8a60e8355bac76333f1f0d10293023c154f
  • Audit of construct, factor, cluster, bootstrap, data, code, and reproducibility: reports/verification/article-348-pinocchio-dimension-construct-factor-cluster-bootstrap-data-code-and-reproducibility-audit.json