Synthetic Personality Review

AI Psychometrics: Assessing the Psychological Profiles of Large Language Models Through Psychometric Inventories - PMC

Evaluation and psychometric validity2024PubMed CentralApproved editorial review

Original title: AI Psychometrics: Assessing the Psychological Profiles of Large Language Models Through Psychometric Inventories

Authors: Max Pellert, Clemens M Lechner, Claudia Wagner, Beatrice Rammstedt, Markus Strohmaier

Keywords: Artificial intelligence, Psychometrics, Natural language processing, Personality, Values, Moral foundations, Gender diversity beliefs

Source: Open primary source (opens in a new tab)

5
Authors
6
Findings
6
Limitations
7
Evidence

Editorial summary

English

This perspective article defines “AI psychometrics” as the critical reuse of psychological inventories to diagnose noncognitive characteristics that language models express or mimic, and supports the proposal with demonstrations on open natural-language-inference models. Rather than eliciting free text from generative systems, it treats each inventory item as a premise and each verbal response option as a hypothesis, records entailment probabilities, selects the maximum-probability option, and aggregates items with the inventory's original scoring rules. Seven unique RoBERTa-, DeBERTa-, BERT-, and BART-family models are involved, with six English configurations and three German configurations. The instruments cover the Big Five, Dark Tetrad, Schwartz values, moral foundations, and beliefs about gender and sex diversity. English Big Five profiles are fairly homogeneous: high Agreeableness and Extraversion and low Neuroticism, with more variation in Openness and Conscientiousness; German profiles diverge more. Dark Tetrad scores generally fall between 2 and 3 on the five-point scale rather than at pathological extremes. On the PVQ-RR, some models barely differentiate the ten values, and DeBERTa shows the clearest pronoun-linked gap for Achievement. Relative to a published reference for politically moderate Americans, models place more weight on Authority, In-group Loyalty, and Purity. On the GSDB, all emphasize Uniformity and show weak Affirmation of gender-diverse identities. These are descriptive outputs of one method and set of snapshots, not evidence that models possess psychological traits or that human inventories are valid for machines. The authors identify reliability, validity, temporal stability, cross-language comparison, downstream behavior, adversarial robustness, and longitudinal monitoring as unresolved research needs. They also explicitly frame the human analogy as metaphorical: model language patterns can have real deployment consequences, but predictive systems do not share human mental life, physiology, or embodied context.

Español

Este artículo de perspectiva define «psicometría de IA» como el uso crítico de inventarios psicológicos para diagnosticar rasgos no cognitivos que los modelos expresan o imitan, y acompaña la propuesta con demostraciones en modelos abiertos de inferencia de lenguaje natural. En vez de pedir texto libre a modelos generativos, presenta cada ítem como premisa y cada opción verbal de respuesta como hipótesis; registra la probabilidad de entailment, elige la opción de máxima probabilidad y agrega los ítems según las reglas originales. Se estudian siete modelos únicos de las familias RoBERTa, DeBERTa, BERT y BART, seis en inglés y tres configuraciones en alemán. Los instrumentos cubren Big Five, tétrada oscura, valores de Schwartz, fundamentos morales y creencias sobre diversidad de género y sexo. Los perfiles Big Five ingleses resultan bastante homogéneos: alta Agreeableness y Extraversion y bajo Neuroticism, con más diferencias en Openness y Conscientiousness; las versiones alemanas divergen más. Las puntuaciones de la tétrada oscura se sitúan generalmente entre 2 y 3 sobre 5, sin perfiles extremos. En el PVQ-RR algunos modelos diferencian poco los diez valores y aparece una diferencia por pronombre especialmente visible en Achievement para DeBERTa. Frente a una referencia de estadounidenses políticamente moderados, los modelos ponderan más Authority, In-group Loyalty y Purity. En la escala GSDB todos enfatizan Uniformity y muestran poca Affirmation de identidades diversas. Estos son resultados descriptivos del método y las instantáneas elegidas, no pruebas de que los sistemas posean rasgos psicológicos ni de que las escalas sean válidas para máquinas. Los autores subrayan como agenda pendiente la fiabilidad, validez, estabilidad temporal, comparación entre idiomas, relación con conducta posterior, robustez adversarial y seguimiento de nuevas versiones. También advierten que la analogía con humanos es metafórica: los modelos son sistemas predictivos cuyos patrones lingüísticos pueden tener consecuencias reales al desplegarse, pero no comparten la vida mental, fisiología o contexto humano.

Research question

How can psychometric inventories of non-cognitive traits be reused as diagnostic tools for LLMs, what do demonstrations through zero-shot classification reveal, and what conceptual and validity problems must AI psychometrics solve?

Method

Perspective article with descriptive demonstrations. Each inventory item is introduced as a premise in models adjusted for natural language inference; each verbal label of the scale is tested as a hypothesis and selected by argmax of the entailment probability. Responses are scored with the original rules. Six configurations in English and three in German from seven unique models are compared, without stochastic sampling, human participants, or inferential tests of validity or reliability.

Sample: Seven unique models available on Hugging Face; six are evaluated in English and XLM-RoBERTa, multilingual DeBERTa, and GBERT in German. Each model responds once per entailment probability to all applicable items; there are no human participants, generative repetitions, or probabilistic sampling of models.

Findings

  • The six Big Five profiles in English are similar: high Agreeableness and Extraversion, low Neuroticism, and more visible differences in Openness and Conscientiousness; the three German profiles show greater variation.
  • Most scores for Machiavellianism, Narcissism, Psychopathy, and Sadism fall approximately between 2 and 3 out of 5; DistilRoBERTa and BART stand out for higher Narcissism, without the study finding extreme profiles.
  • On the PVQ-RR, multilingual DeBERTa and BART differentiate little between values; DeBERTa assigns high Achievement and shows the largest observed difference between the version with masculine and feminine pronouns.
  • The models place more emphasis than the moderate American reference on Authority-respect, In-group Loyalty, and Purity-sanctity, foundations associated in the cited literature with conservative orientations.
  • On GSDB, all models emphasize the uniformity of people of the same gender or sex and show little affirmation of diversity, including disagreement with several statements about non-binary identities.
  • The method avoids the stochastic variation and example ordering typical of some generative prompts, but the results are limited to entailment probabilities of NLI models.

Limitations

  • The article is primarily programmatic and the demonstrations are descriptive: it does not calculate internal consistency, test-retest, convergent, discriminant, factorial, or predictive validity for the scales applied to models.
  • The selection is restricted to open encoder/NLI models and excludes GPT-3, GPT-4, and ChatGPT; therefore, it does not represent the functioning of generative assistants or autoregressive models on free text.
  • Argmax reduces each distribution to a single option and does not examine uncertainty, sensitivity to formulation, contradiction between instruments, variation by prompt, or stability across runs and versions.
  • The English and German comparisons mix language, translation, corpus, and model; they do not allow attributing differences to culture or demonstrating measurement invariance.
  • Some human comparisons use published references from specific groups, not a human sample collected under the same protocol, and the scales were chosen illustratively.
  • It is not tested whether the profiles predict decisions, text generation, tool use, or harms in a real application.

What the study does not establish

  • It does not demonstrate that the models have personality, values, morality, beliefs, motivations, or subjective experience; the terms are used metaphorically for output patterns.
  • It does not psychometrically validate BFI, SD4, PVQ-RR, MFQ, or GSDB for LLMs, nor does it make their scores equivalent to those of humans.
  • It does not demonstrate that a model is politically conservative, transphobic, or harmful in general; it identifies responses from these configurations to specific instruments.
  • It does not allow inferring that differences between language versions are cultural differences or that they arise from a specific component of the training data.
  • It does not establish that modifying a psychometric score will causally and safely modify the subsequent behavior of the model.

Models evaluated

  • XLM-RoBERTa large XNLI
  • DistilRoBERTa base MNLI
  • DeBERTa base MNLI
  • multilingual DeBERTa v3 base MNLI/XNLI
  • GBERT German zero-shot
  • BART large MNLI
  • DistilBART MNLI 12-1

Instruments and metrics

  • Big Five Inventory, 44-item English and 45-item German versions
  • Short Dark Tetrad, 28 items
  • Revised Portrait Values Questionnaire, 57 items and male/female pronoun versions
  • Moral Foundations Questionnaire
  • Gender/Sex Diversity Beliefs Scale
  • Zero-shot natural-language-inference entailment scoring

Data used

  • SNLI, MultiNLI and XNLI fine-tuning corpora underlying the selected models
  • Published human reference scores for selected inventories

Evidence and location

  • Thesis, metaphorical definition, and scope of AI psychometrics: Paper, pp. 808–811, Abstract through Opportunities for Psychometric Assessment
  • Zero-shot NLI method and included models: Paper, pp. 811–812 and 819–822, Figure 1, Appendix Methods and Table A1
  • Big Five and Dark Tetrad results: Paper, pp. 811–813 and 821, Figures 2 and A1
  • Values, pronouns, and moral foundations: Paper, pp. 813–815, Figures 3–4
  • Beliefs about gender and sex diversity: Paper, pp. 816–817, Figure 5
  • Open problems of reliability, stability, and subsequent behavior: Paper, pp. 817–819, Open Challenges and Conclusions
  • Caution against anthropomorphization and limits of the analogy: Paper, pp. 818–819, Conclusions