Synthetic Personality Review

Machine individuality: Separating genuine idiosyncrasy from response bias in large language models

Evaluation and psychometric validity2026arXivApproved editorial review

Authors: Valentin Kriegmair, Dirk U. Wulff

Keywords: Psychometrics, Safety and bias

Source: Open primary source (opens in a new tab)

2
Authors
9
Findings
17
Limitations
4
Evidence

Editorial summary

English

Kriegmair and Wulff ask whether different LLMs show stable, stimulus-specific differences that cannot be reduced to semantic consensus, a global scale offset or sampling noise. Ten open-weight models rated 107,083 words on 14 psycholinguistic norms, nominally five times at temperature 1, yielding about 74.9 million valid ratings. A mixed model separates word effect, model offset, model-by-word interaction and residual. The interaction, termed machine individuality, accounts for 16.9% of variance on average and ranges from 4.8% to 34.0%; additive simulations without an interaction yield much smaller values. Predicting one norm's BLUPs from the other 13 gives higher R-squared when predictors and target come from the same model, consistent with coherent lexical fingerprints. This is meaningful evidence of stable structure in word judgments, but it is not equivalent to personality: the model removes an additive offset, not every nonlinear response style, and it does not test contextual persistence or downstream behavior. The human-alignment comparison also confounds temperature and aggregation because it contrasts a mean of five stochastic samples with one deterministic response. Substantial clean data are public, but BLUPs and results are missing, the pipeline fails on an undefined variable and the required human file is absent, preventing clean end-to-end reproduction.

Español

Kriegmair y Wulff preguntan si distintos LLM muestran diferencias estables y específicas del estímulo que no se reduzcan al consenso semántico, a un desplazamiento global de escala o al ruido de muestreo. Diez modelos abiertos puntuaron 107.083 palabras en 14 normas psicolingüísticas, nominalmente cinco veces a temperatura 1, lo que produjo unos 74,9 millones de ratings válidos. Un modelo mixto separa efecto de palabra, offset de modelo, interacción modelo×palabra y residual. La interacción, denominada individualidad de máquina, representa en promedio 16,9% de la varianza y va de 4,8% a 34,0%; simulaciones aditivas sin interacción producen valores mucho menores. BLUPs de una norma predichos desde las otras 13 muestran mayor R² cuando predictores y objetivo pertenecen al mismo modelo, compatible con huellas léxicas coherentes. Este resultado es importante como evidencia de estructura estable en juicios de palabras, pero no equivale a personalidad: el modelo elimina un offset aditivo, no todos los estilos de respuesta no lineales, y no prueba persistencia contextual ni conducta posterior. La comparación de alineamiento humano también confunde temperatura y agregación, porque contrasta la media de cinco muestras estocásticas con una sola determinista. Los datos limpios son públicos, pero faltan BLUPs y resultados, el pipeline falla por una variable indefinida y no incluye el fichero humano requerido, impidiendo una reproducción limpia integral.

Research question

Do stable and word-specific differences exist between LLMs that exceed shared consensus, a directional offset per model, and stochastic noise, and do those differences form coherent traces across semantic domains?

Method

Ten open LLMs rate a common vocabulary of 107,083 words on 14 psycholinguistic scales using numerical zero-shot prompts. One deterministic response and up to five stochastic responses are collected per cell. For each norm, a crossed mixed linear model estimates intercept, random word effect, model effect, word×model interaction, and residual. One hundred parametric simulations per norm generate data without interaction. Coherence is evaluated with Ridge and five-fold cross-validation: the BLUPs of the held-out norm are predicted from the other 13 and within-model R² is compared with predictors from other models. Human alignment uses Pearson correlations with published norms.

Sample: Ten models, 107,083 words, and 14 norms. The target design contains 1,499,162 model-norm-word cells and up to five stochastic repetitions per cell, approximately 74.9 million valid observations. Human norms cover unequal subsets, from 751 words for morality to 37,056 for concreteness.

Findings

  • The model×word interaction represents on average 16.9% of the total variance, with 4.8% in valence and 34.0% in gender association; the shared word effect averages 32.1%.
  • In one hundred simulations per norm without interaction, the maximum spurious variance remains below 0.033% and the 14 contrasts reach the minimum possible p with correction, 1/101 ≈ 0.0099.
  • The mean specificity ratios per model range from 1.74 to 3.43 for interaction BLUPs, higher than the raw ratings ratios, 1.16 to 1.49.
  • Cross-prediction by words indicates that the deviations of one norm contain shared structure with other norms within the same model.
  • The aggregated correlations with human norms lie between 0.48 and 0.67, showing partial agreement, not equivalence with human representation.
  • The mean of five stochastic responses improves human r by 0.032 over a single deterministic response across the ten models and the 14 norms, but the contrast simultaneously changes temperature and number of samples.
  • The complete audited Qwen file pair retains nearly all cells: two deterministic ratings and 174 stochastic repetitions are missing after filtering.
  • The cleaned files retain invalid attempts alongside replacements; in Qwen, attempt_type appears as zero_shot for all rows even though retry backends and elevated temperatures exist.
  • The central result describes model-specific lexical traces under fixed prompts and scales; the article itself leaves contextual persistence and behavioral prediction as open questions.

Limitations

  • The independent unit of generalization across systems is an intentional set of only ten models, even though the number of ratings is enormous.
  • The model effect has ten levels and each family, organization, architecture, template, tokenizer, quantization, and configuration may contribute to the observed trace.
  • The LMM removes an additive offset per model, but not slopes, compression, use of extremes, or model-specific nonlinear transformations; those styles may enter into model×word.
  • An isolated word is not a behavioral situation; applying the 14 scales to the entire vocabulary includes rare terms, fragments, insults, and unnatural construct combinations.
  • The null simulation validates against a fitted additive Gaussian generator, not against scale errors, tokenization, adapters, retries, or alternative response styles.
  • Specificity reserves words, not models, families, or providers; it does not demonstrate that the trace identifies or predicts a future model.
  • R² ratios hide numerators and denominators; the BLUPs and exact outputs are not published to check stability when the cross R² is small.
  • The human summary averages Fisher-z per norm with equal weight despite very different overlap sizes and related norm families.
  • The supposed reproducibility-alignment trade-off compares five averaged stochastic samples with a single deterministic one and does not isolate temperature from noise reduction.
  • The submission script includes Nomos-1 and the registry also includes Llama 3.1 8B, but neither appears in data or paper and their exclusion is not documented.
  • The 20B-235B range of the method contradicts Phi-4 at 14B; the SI also mixes total and active parameters when citing a minimum of 9B.
  • Models and tokenizers do not fix revisions; the vLLM container does not fix a digest, the order is shuffled without a seed, and there is no lockfile.
  • OSF publishes 2.76 GB of cleaned CSVs, but not BLUPs, RDS models, simulations, tables, figure, human alignment outputs, or psychNorms.csv.
  • The documented command --test fails at step 0 because it uses REPO_ROOT without definition; OSF is also not downloaded or placed automatically in the expected paths.
  • Parallel blocks convert norm failures into messages and may continue, and the audit_completeness validator is not part of the pipeline before declaring success.
  • The CSVs retain endpoint paths from the inference infrastructure, metadata unnecessary for replication that were not redacted.
  • The OSF node lacks a description and its own license; GitHub declares CC BY 4.0, but the scope over the data file is not made explicit.

What the study does not establish

  • It does not demonstrate personality, consciousness, agency, identity, persistent self, or individuality comparable to human.
  • It does not demonstrate that the model×word interaction is free of all response biases, scales, or lexical artifacts.
  • It does not establish that the traces persist in contextualized dialogues, other prompts, languages, versions, or weight updates.
  • It does not demonstrate that the differences predict decisions, safety, persuasion, applied morality, or deployed behavior.
  • It does not identify architecture, size, organization, or training as causes of the differences among the ten models.
  • It does not prove that deviating from human norms implies sui generis representations; corpus, tokenization, instructions, and calibration may also explain it.
  • It does not establish a causal advantage of stochastic sampling over deterministic decoding while keeping the number of observations constant.
  • It does not offer a comprehensive executable reproduction from a clean clone with the public artifacts and the documented command.

Traceability

Scope: Full text

Version: arXiv:2604.16755v1

Consulted source: https://arxiv.org/abs/2604.16755

Review: Codex 17-page visual full-text, TeX/SI, repository, OSF manifest, complete Qwen-file, construct, statistics, code and reproducibility audit, 2026-07-17

Approval: Codex fidelity pass, 2026-07-17

English translation: approved, 2026-07-18

Models evaluated

  • Qwen3-32B
  • Qwen3-235B-A22B-Instruct-2507
  • Mistral-Small-24B-Instruct-2501
  • Gemma 3 27B IT
  • GPT-OSS 20B
  • GPT-OSS 120B
  • OLMo 3.1 32B Instruct
  • Falcon H1 34B Instruct
  • Granite 4.0 H Small
  • Phi-4

Instruments and metrics

  • 14 normas psicolingüísticas de psychNorms
  • Modelo lineal mixto cruzado
  • Bootstrap paramétrico
  • Ridge con validación cruzada
  • Correlación con normas humanas

Data used

  • psychNorms metabase
  • Vocabulario común de 107.083 palabras
  • OSF T9S3M: 20 CSV limpios comprimidos

Evidence and location

  • Design, results, mixed model, simulation, specificity, human correlation, limits, and prompts: arXiv:2604.16755v1, 17 rendered and inspected pages; main TeX and complete SI
  • Generation, retries, LMM, Ridge, alignment, simulation, pipeline, and license: valentinkm/MachineIndividuality commit 8f37d156c5a7c357437357e797cdb09740e92090
  • Inventory of 20 files and complete audit of the deterministic/stochastic Qwen pair: OSF T9S3M manifest SHA-256 255fccb4b8ce5ca47c1fa45f37fddf7e40310341a8d91d7a237e524709fc095d
  • Construct, statistical, data, code, security, and reproducibility audit: reports/verification/article-362-machine-individuality-construct-random-effects-data-code-and-reproducibility-audit.json