Synthetic Personality Review

Large Language Model Psychometrics: A Systematic Review of Evaluation, Validation, and Enhancement

Reviews, theory, and governance2025arXivApproved editorial review

Authors: Haoran Ye, Jing Jin, Yuhang Xie, Xin Zhang, Guojie Song

Keywords: LLM Psychometrics, Systematic Review, Evaluation, Validation

Source: Open primary source (opens in a new tab)

5
Authors
12
Findings
15
Limitations
13
Evidence

Editorial summary

English

This review builds a broad framework for what it calls “LLM Psychometrics”: applying psychometric instruments, theories, and measurement principles to models themselves in order to evaluate, interpret, and modify their behavioral patterns. Its ontological caution is essential. Personality, values, morality, or intelligence describe observable synthetic manifestations, not claims of subjective experience, consciousness, or human psychological states. The framework organizes evaluation around three questions, what construct to measure, how to measure it, and how well the result is validated, and adds a fourth: how measurement can inform system improvement.

The map covers non-cognitive constructs, personality traits, values, morality, and attitudes, and cognitive constructs, heuristics, theory of mind, emotional and social intelligence, psycholinguistics, learning, and reasoning. It distinguishes structured tests, open-ended conversations, and agentic simulations; established inventories, curated items, and synthetic data; persona prompts, performance-enhancing prompts, and adversarial perturbations; and closed, probabilistic, lexical, human, or model-based scoring. Rather than assuming that a plausible score is interpretable, a central section addresses reliability, content validity, construct equivalence, response sets, social desirability, criterion and ecological validity, standardization, and norming.

The synthesis reports heterogeneous and often contradictory evidence. Some models show high internal consistency or prosocial profiles on questionnaires, but those profiles shift with prompts, option order, language, format, context, and model version. Direct self-reports can diverge from decisions or open conversations; human instruments may not recover the same factor structure; and correct answers may arise from memorization or shortcuts rather than the cognitive process an item is intended to measure. The review recommends AI-specific instruments, uncontaminated stimuli, parallel forms, exact inference reporting, factor analysis and IRT, external criteria, and ecological tasks. It also maps how scales or constructs are used for steering, fine-tuning, alignment rewards, and social or cognitive enhancement, although it does not estimate the general effectiveness of those interventions.

The document is an exceptionally extensive conceptual reference, 57 pages and 412 bibliography entries, but its “systematic review” label is not supported by a reproducible review method. It reports no searched databases, query, search dates, complete operational criteria, record counts, deduplication, screening, extraction process, PRISMA flow, quality appraisal, or risk-of-bias assessment. Table 2 assigns “supporting” or “contradicting” models to narrative findings without weighting design, sample size, dependence, uncertainty, or psychometric quality. The repository provides a useful reading list but not a structured corpus or inclusion history; it has no license, and its embedded PDF is an older May 2025 version while the reviewed text is v3 from March 2026. The work is therefore a valuable critical treatise and taxonomy, not an exhaustive evidence base, meta-analysis, or unbiased estimate of field consensus.

Español

Esta revisión construye un marco amplio para lo que denomina «psicometría de LLM»: aplicar instrumentos, teorías y principios de medición psicológica a los propios modelos para evaluar, interpretar y modificar sus patrones de conducta. Su precaución ontológica es esencial: personalidad, valores, moralidad o inteligencia se usan para describir manifestaciones sintéticas observables, no para afirmar experiencia subjetiva, conciencia o estados psicológicos humanos. El marco organiza la evaluación mediante tres preguntas, qué constructo medir, cómo medirlo y con qué validez, y añade una cuarta: cómo utilizar esa medición para mejorar el sistema.

El mapa cubre constructos no cognitivos, rasgos de personalidad, valores, moral y actitudes, y cognitivos, heurísticos, teoría de la mente, inteligencia emocional y social, psicolingüística, aprendizaje y razonamiento. Distingue tests estructurados, conversaciones abiertas y simulaciones de agentes; inventarios establecidos, ítems curados y datos sintéticos; prompting de personas, ayudas al rendimiento y perturbaciones adversarias; y puntuación cerrada, probabilística, léxica, humana o mediante otro modelo. En vez de asumir que una puntuación plausible es interpretable, dedica un bloque central a fiabilidad, validez de contenido, equivalencia de constructo, sesgos de respuesta, deseabilidad social, validez criterial y ecológica, estandarización y baremación.

La síntesis presenta evidencia heterogénea y a menudo contradictoria. Algunos modelos muestran alta consistencia interna o perfiles prosociales en cuestionarios, pero esos perfiles cambian con prompts, orden de opciones, idioma, formato, contexto y versión. Las respuestas directas pueden divergir de decisiones o conversaciones abiertas; los tests humanos pueden no recuperar la misma estructura factorial; y un acierto puede proceder de memorización o atajos en lugar del proceso cognitivo que el ítem pretende medir. La revisión recomienda instrumentos específicos para IA, estímulos no contaminados, variantes paralelas, documentación exacta de inferencia, análisis factorial e IRT, criterios externos y tareas ecológicas. También muestra cómo escalas o constructos se usan para steering, fine-tuning, recompensas de alignment y mejora social o cognitiva, aunque la eficacia general de esas intervenciones no se estima.

El documento es una referencia conceptual excepcionalmente extensa, 57 páginas y 412 entradas bibliográficas, pero su etiqueta de «systematic review» no queda respaldada por un método de revisión reproducible. No publica bases consultadas, consulta, fechas de búsqueda, criterios operativos completos, número de registros, deduplicación, cribado, extracción, diagrama PRISMA, evaluación de calidad o riesgo de sesgo. La Tabla 2 asigna modelos «supporting» o «contradicting» a conclusiones narrativas sin ponderar diseño, tamaño, dependencia, incertidumbre o calidad psicométrica. El repositorio aporta una lista útil, pero no un corpus estructurado ni el historial de inclusión; carece de licencia y su PDF embebido es una versión antigua de mayo de 2025, mientras el texto revisado es v3 de marzo de 2026. Por ello, la obra sirve como tratado y taxonomía crítica de alto valor, no como prueba exhaustiva, metaanálisis ni estimación imparcial del consenso del campo.

Research question

How can psychometric instruments and principles reorganize the evaluation of LLMs, from psychological constructs and methodology to validation, and how can those measurements guide the control, alignment, and improvement of models?

Method

Narrative synthesis of literature grouped into: differences between psychometrics and benchmarking; personality and cognition constructs; formats, data, prompts, outputs, and scoring; reliability and validity; enhancement; trends and ethics. The declared scope includes works that treat the LLM as a subject and employ psychometric instruments or principles, including those that only administer scales and report scores; it excludes the use of the LLM as a tool to measure people and purely scalar benchmarks. It is not documented how the literature was searched, selected, extracted, or appraised. The editorial review read the 35 pages of content, checked the 22 pages of references, the official site, the arXiv v3 version, and the curated repository.

Sample: There is no sample of models or participants analyzed directly. The bibliography lists 412 references and the text synthesizes families of studies published mainly up to 2025, but those 412 entries include foundations, instruments, blogs, surveys, and empirical works; they do not equate to 412 included studies. The manuscript does not report the size of the review corpus, records identified or excluded, languages, databases, time window, or distribution by construct. The public list contains hundreds of links and duplicates between labels, but offers no IDs, normalized metadata, or inclusion decisions.

Findings

  • Psychometrics seeks to define and validate latent constructs, item quality, reliability, and predictive power; AI benchmarking usually prioritizes breadth, difficulty, ranking, and aggregate task metrics.
  • The legitimate unit of interpretation proposed is the synthetic behavioral manifestation in outputs, not an attribution of consciousness, subjective experience, or internal human psychology.
  • The field covers personality, values, morality, and opinions, along with biases, social cognition, language, learning, and reasoning; each construct produces results dependent on the model and protocol.
  • Structured tests are scalable and standardizable but suffer from contamination, response biases, and low ecological validity; conversations and simulations gain realism at the cost of scoring, control, and reproducibility.
  • The same LLM can be treated as an individual, as a population induced through profiles, or through a hybrid approach of variations of the same test; each choice measures a distinct object and requires distinct validation.
  • High internal consistency is not sufficient: reliability varies across repetition, parallel forms, languages, option order, judges, attacks, and formats, and factorial model validity can fail.
  • Human-LLM equivalence cannot be assumed. Big Five, HEXACO, and human value systems may not recover the same dimensions; AI-native proposals emerge, but they do not yet converge on a single structure either.
  • Acquiescence, option position, scale extremes, and social desirability can generate apparently coherent profiles without the item content determining the response.
  • Self-report scores frequently diverge from conversations, decisions, or downstream tasks, which makes criterion and ecological validity insufficient for much of the literature.
  • IRT, factor analysis, procedural item generation, parallel forms, novel stimuli, and explicit norms offer ways to better measure difficulty, discrimination, contamination, robustness, and comparability.
  • Psychometrics is used to control traits, design reward signals, guide alignment, and improve empathy or reasoning, but these lines remain early and there is no general estimation of benefit or cost.
  • Standardization and norming remain open: there is no established reference population for models that change rapidly and whose profiles depend on the prompt.

Limitations

  • The manuscript calls itself a systematic review, but does not report databases, search strings, dates, complete criteria, records, duplicates, reviewers, extraction, PRISMA, or a preregistered protocol.
  • There is no evaluation of quality, certainty, risk of bias, or peer review status; exploratory studies, preprints, benchmarks, and works with rigorous validation are synthesized at the same narrative level.
  • The 412 references do not constitute an identifiable corpus of included studies: they mix empirical literature, psychometric theory, instruments, surveys, commentary, web pages, and contextual works.
  • Table 2 uses lists of supporting/contradicting models as a substitute for evidence synthesis, without distinguishing independent studies, multiple measurements from the same work, size, version, prompt, inference, or uncertainty.
  • There is no meta-analysis, effect sizes, intervals, sensitivity, heterogeneity analysis, or publication bias; "convergent evidence" and "consistently" are unquantified editorial judgments.
  • Comparing models by family or size confounds date, training data, post-training, product policies, inference parameters, and instrument selection. It does not allow causal attribution of more prosocial profiles or greater alignment to scaling.
  • Expressions such as "desirable human values" require a normative position and reference population; the review recognizes cultural pluralism, but some syntheses reduce that diversity to a single desirable direction.
  • The personality section claims psychometric validity for advanced models relying on specific studies, while other reviewed works find invalid factorial structures; the scope of the claim should be limited to specific protocols.
  • The framework groups different constructs, capabilities, and phenomena under LLM psychometrics. The relationship between a statistical pattern of output, an internal mechanism, and downstream utility remains unresolved.
  • Scoring by another LLM may share biases, errors, or preferences with the evaluated system; agreement among model-judges does not automatically equate to human or external validity.
  • Human results used as norms may not be comparable by language, culture, mode of administration, and consent; the review itself recognizes contamination, copyright, and secondary reuse of data.
  • The rapid updating of models renders many conclusions obsolete. The v3 version is from March 2026, but the bibliography mainly extends to 2025 and does not define a cutoff date.
  • The public list does not include a structured bibliographic file, search traceability, inclusion decisions, hashes, or link QA; it contains repeated entries between categories and lacks a license.
  • The PDF included in the repository was created in May 2025 and does not match the reviewed arXiv v3 version; the last audited commit is from November 2025.
  • The reviewed document is an arXiv preprint; the manuscript and the site do not identify a final peer-reviewed publication.

What the study does not establish

  • It does not demonstrate that LLMs possess personality, values, morality, intelligence, or theory of mind as internal human psychological states.
  • It does not demonstrate that human constructs are equivalent in machines; it precisely identifies equivalence as a hypothesis to be tested.
  • It does not demonstrate that a high score, a high alpha, or a correct response are valid without adequate structure, process, robustness, and external criterion.
  • It does not establish a universal psychological profile of LLMs: results depend on family, snapshot, prompt, format, language, sampling, and context.
  • It does not prove that larger or more recent models are intrinsically more moral, safe, or representative; observational comparisons are confounded.
  • It does not demonstrate that role-playing generates a valid human population or that prompt-induced variation equates to real individual differences.
  • It does not prove that proposed AI-native systems measure a definitive ontology; different methods discover different, task-dependent structures.
  • It does not demonstrate that psychometric interventions causally improve safety, alignment, or cognition in a general way or that they lack trade-offs.
  • It does not allow ranking models across heterogeneous studies or applying human norms to LLM scores without equivalence validation.
  • It does not constitute an exhaustive or reproducible search of all the literature despite its systematic review label.
  • It does not establish scientific consensus on the sentience, understanding, cognition, or humanity of models.
  • It does not demonstrate that psychometrics alone resolves the evaluation of multimodal systems, agents, or real high-stakes interactions.

Traceability

Scope: Full text

Version: arXiv:2505.08245v3 (11 Mar 2026), 57 pages; official project website reviewed; public collection commit e06a725d195ac6d655fff8d25580b49004b61d15 audited

Consulted source: https://arxiv.org/pdf/2505.08245

Review: Codex editorial review and methods/artifact audit, 2026-07-15

Approval: Codex fidelity pass, 2026-07-15

English translation: approved, 2026-07-18

Models evaluated

  • No model is evaluated directly by the review
  • OpenAI GPT-2, GPT-3, GPT-3.5, GPT-4, GPT-4o, o1 and o3 families across cited studies
  • Meta Llama and Llama 2/3/3.1/3.3 families across cited studies
  • Anthropic Claude families across cited studies
  • Google PaLM, Gemini, Gemma and Bard families across cited studies
  • Mistral and Mixtral families across cited studies
  • Qwen, Yi, DeepSeek, Baichuan, ChatGLM, Falcon, Phi and other open model families
  • Multimodal, agentic and multi-agent systems covered conceptually

Instruments and metrics

  • Big Five inventories: NEO-PI-R, BFI and BFI-2
  • HEXACO-60 and HEXACO-100
  • Dark Triad Dirty Dozen and MBTI
  • Schwartz Value Survey, Portrait Values Questionnaire and World Values Survey
  • Hofstede VSM, GLOBE and Social Value Orientation
  • Moral Foundations Questionnaire, Moral Foundations Vignettes and Defining Issues Test
  • ANES, ATP, GLES, Political Compass and other opinion surveys
  • Cognitive Reflection Test, false-belief tasks, WAIS-IV, Raven matrices and ARC
  • Emotional and social intelligence scales and situational judgment tests
  • Cronbach's alpha, hierarchical omega, test-retest, parallel forms and inter-rater reliability
  • Factor analysis, structural equation modeling, measurement invariance and Item Response Theory
  • Content, procedural, construct, criterion, ecological and cross-lingual validity

Data used

  • No primary experimental dataset (literature review)
  • Bibliography containing 412 numbered references
  • Official Awesome-LLM-Psychometrics curated reading list
  • Established human psychometric inventories and normative datasets surveyed
  • Custom-curated and LLM-generated psychometric item sets surveyed
  • Open-ended dialogue, behavioral and agentic simulation benchmarks surveyed

Evidence and location

  • Version, authorship, scope, and behavioral definition of LLM Psychometrics: arXiv:2505.08245v3, abstract and section 1, pp. 1 and 4-6
  • Differences between psychometrics and benchmarking and integration of ECD, factor analysis, and IRT: arXiv v3, sections 2-3, pp. 6-10
  • Taxonomy of constructs and synthesis of results by models: arXiv v3, section 4, Figures 1-2 and Tables 2-3, pp. 10-21
  • Traits, values, morality, and attitudes, with limits of stability and self-report: arXiv v3, sections 4.1.1-4.1.4, pp. 15-17
  • Biases, social cognition, psychology of language, and capabilities: arXiv v3, sections 4.2.1-4.2.4, pp. 18-21
  • Formats, sources, prompting, scoring, and inference: arXiv v3, section 5, Figures 3 and Tables 4-5, pp. 21-26
  • Reliability, entity versus population, and contradictory results: arXiv v3, section 6.1 and Figure 4, pp. 26-28
  • Content, process, equivalence, response sets, desirability, and ecological validity: arXiv v3, sections 6.2-6.3, pp. 28-30
  • Trait manipulation, safety/alignment and cognitive enhancement: arXiv v3, section 7, pp. 30-31
  • AI-native constructs, anthropomorphization, IRT, standardization, and norming: arXiv v3, section 8, pp. 31-34
  • Consent, copyright, bias, and anthropomorphic attribution: arXiv v3, section 9, p. 34
  • Total bibliography and absence of systematic review method section: Full arXiv v3 manuscript: 412 numbered references; contents and sections 1-10 contain no search, screening or quality-appraisal protocol
  • Public collection, lack of structure/license, and version misalignment: GitHub valuebyte-ai/Awesome-LLM-Psychometrics commit e06a725d195ac6d655fff8d25580b49004b61d15; readme.md and assets/llm_psychometrics_review.pdf