Synthetic Personality Review

Evaluating General-Purpose AI with Psychometrics

Reviews, theory, and governance2026ACMApproved editorial review

Authors: Xiting Wang, Liming Jiang, José Hernández‐Orallo, David Stillwell, Shiqiang Chen, Luning Sun, Fang Luo, Xing Xie

Keywords: Psychometrics, Human evaluation, LLM-as-a-judge, Behavioral alignment

Source: Open primary source (opens in a new tab)

8
Authors
8
Findings
13
Limitations
4
Evidence

Editorial summary

English

Wang and colleagues propose replacing the view of benchmarks as collections of tasks with construct-oriented evaluation: latent dimensions that should predict and explain performance across contexts. The peer-reviewed article is a methodology paper, not a new experiment, and organizes the process into three stages. First, a construct is identified from theory and experts or empirical patterns; next, a test is designed and scored, potentially using IRT, cognitive diagnosis and adaptive testing; finally, the interpretation requires evidence of reliability and construct, convergent, discriminant and predictive validity. Its central warning is especially relevant to personality: administering a human self-report to an LLM does not demonstrate the same trait, because construct-indicator relations may differ and minor prompt or order changes can alter responses. It also leaves open the unit of analysis, what counts as a person and population across prompts, simulated personas and versions, plus sensitivity, alignment faking and human-AI comparison through DIF. The framework is valuable as a safeguard checklist and shared vocabulary, but it does not implement the pipeline, validate a scale, test models or conduct a systematic review. Quantitative examples come from cited work. It also does not operationalize key IRT assumptions, calibration size, local independence, invariance, contamination or model drift. The paper therefore supports requiring validity evidence before claiming personality or capability, not that LLMs possess human traits or that a universal predictive scale already exists.

Español

Wang y colaboradores proponen sustituir la lectura de benchmarks como colecciones de tareas por una evaluación orientada a constructos: dimensiones latentes que deben predecir y explicar rendimiento en múltiples contextos. El artículo, una pieza metodológica revisada por pares y no un experimento nuevo, organiza el proceso en tres fases. Primero se identifica el constructo desde teoría y expertos o desde patrones empíricos; después se diseña y puntúa el test, con herramientas posibles como IRT, diagnóstico cognitivo y evaluación adaptativa; finalmente se exige evidencia de fiabilidad y de validez de constructo, convergente, discriminante y predictiva. Su advertencia central es especialmente relevante para personalidad: aplicar a un LLM un autoinforme humano no demuestra el mismo rasgo, porque pueden cambiar la relación entre constructo e indicadores y variaciones menores del prompt u orden alteran las respuestas. También deja abiertos problemas de unidad de análisis, qué cuentan como persona y población entre prompts, personas simuladas y versiones, sensibilidad, alignment faking y comparación humano-IA mediante DIF. El marco es valioso como lista de garantías y lenguaje común, pero no implementa el pipeline, no valida una escala, no prueba modelos ni realiza revisión sistemática. Los ejemplos cuantitativos proceden de trabajos citados. Tampoco operacionaliza supuestos críticos de IRT, tamaño/calibración, independencia local, invariancia, contaminación o deriva de modelos. Por ello respalda exigir validez antes de hablar de personalidad o capacidad, no que los LLM posean rasgos humanos ni que exista ya una escala universal y predictiva.

Research question

How can psychometrics convert generalist AI evaluation into construct-oriented measurement with predictive and explanatory power and quality controls, and what errors should be avoided when adapting human tests?

Method

Conceptual and methodology article that narratively synthesizes examples of prior research and proposes a three-stage framework: top-down or bottom-up identification of constructs; measurement through item design, specifications, and theories such as IRT; and validation through reliability and distinct forms of validity evidence. It adds open questions about prompts, alignment faking, persona/population, human-AI comparability, and use of DIF, and it outlines a conceptual pipeline to integrate evaluation across AI development.

Sample: There is no own experimental sample. The documentary unit consists of selected examples from prior literature; the article does not declare a search strategy, inclusion criteria, number of screened studies, or quality assessment, so it is a narrative synthesis and not a systematic review.

Findings

  • The framework differentiates three objectives: predicting performance beyond observed tasks, explaining variation through constructs, and ensuring measurement quality with reliability and validity.
  • Identification can start from theory and expert consensus or from empirical structure; sharing a human label does not imply sharing definition, indicators, or mechanism.
  • IRT, adaptive testing, and cognitive diagnosis are presented as potential tools for comparing systems and items, not as a universally validated solution by this work.
  • Validation must combine stability and consistency with evidence that the test measures the construct and predicts relevant criteria; an aggregated score alone is not sufficient.
  • The article expressly questions human self-reports of personality in LLMs: minor changes to the input can alter responses, and the relationship with user perception or interaction quality may be weak.
  • Prompt, simulated persona, fine-tuning, and version complicate what is considered a person, a repetition, and a population; the authors suggest studying variances and using multilevel models.
  • DIF is proposed to check whether an item functions comparably between humans and AI systems, rather than assuming equivalence.
  • The final pipeline integrates objective definition, selection and training with feedback, and subsequent validation, but it remains a diagram and conceptual guide without public implementation.

Limitations

  • It is a position and methodology article: it does not empirically prove that the complete framework outperforms conventional benchmarks.
  • It is not a systematic review; the examples are selective and there is no search, screening, bias assessment, or evidence grading.
  • The cited numerical results belong to other studies and are not reproduced or reanalyzed in this article.
  • The assumption that a few constructs organize much of AI behavior is a useful hypothesis, not a demonstrated property for all tasks or systems.
  • A statistical factor summarizes covariation, but does not by itself establish a natural ability, causal mechanism, or psychological trait.
  • IRT is recommended without converting its assumptions of dimensionality, local independence, monotonicity, fit, identification, and calibration size for few models and many correlated items into a protocol.
  • No mandatory sequence of invariance across languages, families, versions, prompts, and modalities is defined; DIF appears as a possibility for human-AI comparisons.
  • The work identifies but does not resolve what is a person, population, or repetition when prompt, system, tools, context, decoding, or provider change.
  • It mentions item exposure, but does not specify contamination detection, safe custody, scale linking after leaks, or version traceability.
  • External criteria for safety, values, bias, or toxicity are normative and sociotechnical; psychometrics alone does not decide which results or thresholds should govern.
  • The operational pipeline describes a conceptual process; no toolkit, schema, test template, comprehensive case, acceptance criteria, or executable implementation is published.
  • There is no empirical artifact to reproduce; verifying the strength of its examples requires separately auditing each cited study.
  • The open accepted version is later and more complete than arXiv v2; the ACM registration version was not downloadable in this audit due to HTTP 403.

What the study does not establish

  • It does not demonstrate that an LLM has personality, intelligence, emotions, values, mind, or experience equivalent to humans.
  • It does not validate any human questionnaire as a measure of AI traits, nor does it authorize using their names without redefinition and specific evidence.
  • It does not demonstrate that self-report responses are stable, sincere, or represent an internal state.
  • It does not create a universal psychometric scale or a definitive taxonomy of AI capabilities, risks, or values.
  • It does not prove that IRT, factor analysis, or adaptive testing are appropriate for any benchmark or model regime.
  • It does not establish psychological causality from factors or performance correlations.
  • It does not provide an executable operational pipeline despite using that expression in the abstract.
  • It does not demonstrate regulatory compliance, safety, fairness, or deployment readiness solely by applying psychometric metrics.
  • It does not replace real validation with users, tasks, and consequences of each domain.
  • It does not provide own experimental results; its contribution is a peer-reviewed roadmap and a warning about construct validity.

Traceability

Scope: Full text

Version: Communications of the ACM 69(5), 92-102, accepted manuscript; arXiv:2310.16379v2 inspected as predecessor

Consulted source: https://doi.org/10.1145/3769688

Review: Codex dual 10-page accepted-manuscript and 15-page arXiv visual full-text, source, construct-validity, personality, psychometric-assumption and evidence-boundary audit, 2026-07-17

Approval: Codex fidelity pass, 2026-07-17

English translation: approved, 2026-07-18

Models evaluated

  • No evalúa modelos originales; discute ChatGPT y Gemini como ejemplos de IA generalista
  • GPT-3 y GPT-4 aparecen en ejemplos conceptuales
  • OpenAI o4 y o4-mini aparecen al discutir versiones y poblaciones
  • Los resultados cuantitativos sobre LLM pertenecen a estudios citados

Instruments and metrics

  • Identificación top-down y bottom-up de constructos
  • Método Delphi
  • Análisis factorial
  • Teoría de Respuesta al Ítem, IRT
  • Diagnóstico cognitivo y clases latentes
  • Computerized Adaptive Testing
  • Fiabilidad test-retest, consistencia interna, formas paralelas e interjueces
  • Validez de constructo, convergente, discriminante y predictiva
  • Differential Item Functioning
  • Modelado multinivel

Data used

  • No crea ni analiza un dataset original
  • BIG-Bench y Animal-AI Olympics se discuten mediante estudios citados
  • Los ejemplos de factores sobre 29 modelos y 27 tareas, 591 modelos y 500 chatbots proceden de referencias externas

Evidence and location

  • Accepted version, final framework, warnings, opportunities, and references: Communications of the ACM accepted manuscript, DOI 10.1145/3769688, SHA-256 907964f0260d966228f5115c1cef9c9084d7db76f7b3d46d4a665dd3e034441d, 10 pages inspected
  • Complete preprint and prior evolution of the framework: arXiv:2310.16379v2, SHA-256 23ce27d9e585e444206a7ac18753e3a911b02a146a3ffdb8882a37063df9ff16, 15 pages inspected
  • Publication metadata, authorship, volume, issue, and pages: Crossref DOI 10.1145/3769688 and Cambridge repository DOI 10.17863/CAM.121669
  • Audit of article type, construct, personality, psychometric assumptions, operationalization, and evidence boundary: reports/verification/article-365-cacm-psychometrics-construct-framework-personality-validity-operationalization-and-evidence-boundary-audit.json