Synthetic Personality Review

AI Psychometrics: Evaluating the Psychological Reasoning of Large Language Models with Psychometric Validities

Evaluation and psychometric validity2025arXivApproved editorial review

Authors: Yibai Li, Xiaolin Lin, Zhenghui Sha, Zhiye Jin, Xiaobing Li

Keywords: Psychometrics

Source: Open primary source (opens in a new tab)

5
Authors
7
Findings
24
Limitations
5
Evidence

Editorial summary

English

The paper applies PLS-SEM to responses to a thirteen-item Technology Acceptance Model questionnaire about Amazon recommendations. It compares 500 questionnaires from each of four endpoints, GPT-3.5-turbo, GPT-4o, LLaMA-2-13B-chat and LLaMA-3-8B-instruct, with 248 Mechanical Turk participants, and reports that nearly all meet selected convergent- and discriminant-validity thresholds and that newer models produce more human-like structures. However, every synthetic row starts with a randomly imposed 1-7 answer, after which the model completes the other twelve items while seeing the accumulated history: an intervention that induces consistency and is not equivalent to the human procedure. The claimed predictive validity is in-sample R-squared from the same questionnaire, while external validity is reduced to two paths sharing a positive sign. Prompts, responses, human data, code and SmartPLS projects are not released. The tables are also internally irreproducible: LLaMA-2 purchase-intention composite reliability is about 0.689 rather than 0.90 from the published loadings, none of the fifteen Fornell-Larcker diagonals equals the square root of its AVE, and several R-squared values do not follow from the displayed coefficients and correlations. This is best read as a HICSS case study of covariance in randomly anchored sequential completions, not evidence of latent attitudes, human equivalence or general psychological reasoning.

Español

El trabajo aplica PLS-SEM a respuestas de un cuestionario de trece ítems del Modelo de Aceptación Tecnológica sobre recomendaciones de Amazon. Compara 500 cuestionarios por cada uno de cuatro endpoints, GPT-3.5-turbo, GPT-4o, LLaMA-2-13B-chat y LLaMA-3-8B-instruct, con 248 participantes de Mechanical Turk, e informa que casi todos cumplen umbrales de validez convergente y discriminante y que los modelos nuevos producen estructuras más parecidas a las humanas. Sin embargo, cada fila sintética comienza con una respuesta 1-7 impuesta al azar y el modelo completa los otros doce ítems viendo el historial acumulado: una intervención que induce coherencia y no equivale al procedimiento humano. La llamada validez predictiva es R² dentro del mismo cuestionario y la externa se reduce a que dos rutas tengan signo positivo. No se publican prompts, respuestas, datos humanos, código ni proyectos SmartPLS. Además, las tablas no son internamente reproducibles: la fiabilidad compuesta de intención de compra de LLaMA-2 es aproximadamente 0,689 y no 0,90 con las cargas publicadas, ninguna de las quince diagonales Fornell-Larcker coincide con la raíz de su AVE y varios R² no se derivan de los coeficientes y correlaciones mostrados. Debe leerse como un caso de estudio HICSS sobre covarianza en completados secuenciales anclados al azar, no como prueba de actitudes latentes, equivalencia humana o razonamiento psicológico general.

Research question

Do four LLMs produce response structures for the Technology Acceptance Model that satisfy certain PLS-SEM criteria of convergent, discriminant, and structural validity, and do they appear closer to human responses in newer generations?

Method

For each synthetic questionnaire, one item is selected at random, a uniform response between 1 and 7 is externally assigned to it, and the model responds to the remaining twelve in random order conditioned by the entire history of questions and responses. 500 nominal rows are generated per endpoint through OpenRouter. In parallel, 286 Mechanical Turk participants respond in May 2024 and 248 pass the filters. Five separate analyses in SmartPLS estimate loadings, alpha, composite reliability, AVE, Fornell-Larcker, paths, and R², with 5,000 bootstrap resamples.

Sample: Four synthetic samples of 500 completions from a single endpoint and procedure per model, plus 248 final human participants from Mechanical Turk who declared at least two purchases on Amazon during the previous three months. Completions from the same endpoint are not 500 independent models or minds, and synthetic administration with random anchoring and visible history differs from human administration.

Findings

  • GPT-3.5, GPT-4o, LLaMA-3, and the human sample meet the chosen convergent thresholds; LLaMA-2 exhibits weaker loadings and consistency, especially in purchase intention.
  • The article declares that all five samples pass Fornell-Larcker, although the published diagonals are not the square roots of the corresponding AVEs.
  • The reported R² for purchase intention are 18.4%, 44.3%, 19.7%, 37.3%, and 59.9% for GPT-3.5, GPT-4o, LLaMA-2, LLaMA-3, and humans.
  • GPT-4o and LLaMA-3 descriptively show higher loadings, reliability, and R² than their predecessors under this specific procedure.
  • The composite reliability of purchase intention for LLaMA-2 calculated with the four published loadings is approximately 0.689, not 0.90.
  • None of the fifteen published diagonals in the Fornell-Larcker matrices matches the square root of AVE; upon recalculation, the qualitative criterion still holds with the shown correlations.
  • The algebraic identity of the standardized model produces approximate R² of 17.5%, 43.0%, 23.4%, 42.0%, and 45.0%; the larger differences with the table, especially 45.0% versus 59.9% in humans, are not explained by rounding.

Limitations

  • One of the thirteen values in each synthetic row does not come from the model: it is imposed uniformly at random and conditions all subsequent responses.
  • The visible history of items and responses favors semantic coherence within each row and does not reproduce independent administration or the described human procedure.
  • The method called diffusion contains no learned direct/inverse process, noise schedule, or denoiser; it is random anchoring followed by conditional completion.
  • The 500 rows per endpoint are stochastic samples from a shared system, not 500 independently trained subjects or models.
  • The TAM items are semantically redundant and may produce high loadings and alpha through paraphrasing and instruction following, without an internal psychological construct.
  • A single questionnaire on Amazon recommendations does not cover general psychological reasoning, emotions, social cognition, or real behavior.
  • Wording, system prompt, ordering policy, language, provider, and context are not varied, and the preliminary temperature and seed experiment mentioned is not published.
  • No slugs, upstream providers, immutable revisions, execution dates, temperature, top-p, seeds, prompt, output schema, retries, or failures are provided.
  • The 2,000 synthetic questionnaires, the 248 human records, the collection code, the logs, the SmartPLS project, and the exports are not published.
  • Predictive validity is labeled from in-sample R² of the same questionnaire, without holdout, external criterion, future behavior, or observed purchase.
  • External validity is inferred from paths with the same sign within the same survey, without generalization across populations, contexts, times, or instruments.
  • There is no measurement invariance, multigroup comparison, equivalence test, or formal contrast between model generations.
  • Only Fornell-Larcker is reported; HTMT, published cross-loadings, and multimethod evidence are missing.
  • The composite reliability of purchase intention for LLaMA-2 is not reproduced with its loadings and contradicts the claim that all exceed 0.70.
  • The fifteen Fornell-Larcker diagonals contradict the AVEs from the previous table, although the qualitative criterion survives recalculation.
  • Several R² are not derived from the shown path coefficients and correlations; the human discrepancy is especially large.
  • Paths only show stars: standard errors, intervals, exact p-values, and bootstrap configuration beyond 5,000 resamples are missing.
  • Multiple hypotheses are not corrected, and two coefficients being significant separately does not prove they are equal across groups.
  • Demographics, countries, instructions, attention, exclusions by reason, missing data, and representativeness of Mechanical Turk are not described.
  • Ethical review or exemption, consent, participant protection, payment amount, duration, and effective rate are omitted.
  • Thirty-eight of 286 participants disappear from the final sample without a breakdown between dropout, screening, and quality control.
  • The arXiv version appeared in 2026, but the official article was published at HICSS 2025; the official PDF numbers pages 5194-5202 and arXiv cites 5189-5197.
  • Crossref and ScholarSpace attribute the fifth authorship to Emily Lee, while both PDFs and arXiv show Xiaobing Li.
  • There is no limitations section, data/code availability, supplementary appendix, or identifiable repository.

What the study does not establish

  • It does not demonstrate that the models possess the latent attitudes measured by TAM.
  • It does not establish general psychological reasoning from an e-commerce questionnaire with redundant items.
  • It does not provide predictive validity against future behavior or held-out data.
  • It does not provide external validity across populations, contexts, moments, or instruments.
  • It does not demonstrate human equivalence because the procedures differ and invariance and equivalence tests are missing.
  • It does not prove that GPT-4o or LLaMA-3 are generally more valid than their predecessors; it compares one endpoint per generation.
  • It does not offer internally consistent tables of composite reliability, Fornell-Larcker, and R².
  • It does not allow reproduction of the collection or analysis without prompts, responses, data, model paths, code, and SmartPLS artifacts.
  • It does not make the neural network mechanism transparent; PLS-SEM describes covariance among outputs.
  • It does not support conclusions about ethical decisions, alignment with human values, or understanding of thoughts and emotions.

Traceability

Scope: Full text

Version: arXiv:2603.11279v1; HICSS 2025 DOI 10.24251/HICSS.2025.623

Consulted source: https://arxiv.org/abs/2603.11279

Review: Codex 15-page arXiv plus 9-page official HICSS visual full-text, psychometric-validity, sampling, arithmetic, data/code, human-ethics and reproducibility audit, 2026-07-17

Approval: Codex fidelity pass, 2026-07-17

English translation: approved, 2026-07-18

Models evaluated

  • GPT-3.5-turbo mediante OpenRouter
  • GPT-4o mediante OpenRouter
  • LLaMA-2-13B-chat mediante OpenRouter
  • LLaMA-3-8B-instruct mediante OpenRouter
  • SmartPLS para PLS-SEM

Instruments and metrics

  • Modelo de Aceptación Tecnológica de 13 ítems y escala Likert de 7 puntos
  • Cinco ítems de utilidad percibida
  • Cuatro ítems de facilidad de uso
  • Cuatro ítems de intención de compra
  • Cargas factoriales, alfa de Cronbach, fiabilidad compuesta y AVE
  • Criterio Fornell-Larcker
  • Coeficientes de ruta y R² en PLS-SEM

Data used

  • 2.000 cuestionarios sintéticos nominales, 500 por endpoint, no publicados
  • Encuesta Mechanical Turk de 286 participantes iniciales y 248 finales, no publicada
  • Proyecto y exportaciones SmartPLS no publicados

Evidence and location

  • Article, method, instrument, tables, discussion, and references: arXiv:2603.11279v1, all 15/15 PDF pages rendered and individually inspected
  • Published version, DOI, visible authors, license, and official pagination: HICSS 2025 DOI 10.24251/HICSS.2025.623, all 9/9 official proceedings PDF pages rendered and individually inspected
  • Date, pagination, and fifth authorship conflict in metadata: Official ScholarSpace item/bitstream metadata and Crossref DOI metadata inspected 2026-07-17
  • Absence of data, code, and analysis project in public materials: Official arXiv source archive and authenticated GitHub repository/code search inspected 2026-07-17
  • Reproduction of composite reliability, AVE, Fornell-Larcker, and R²; audit of sampling, validity, ethics, and reproducibility: reports/verification/article-393-ai-psychometrics-random-anchor-pseudoreplication-validity-arithmetic-data-human-ethics-and-reproducibility-audit.json