The Emergence of Lab-Driven Alignment Signatures: A Psychometric Framework for Auditing Latent Bias and Compounding Risk in Generative AI

Applications, bias, and safety2026arXivApproved editorial review

Authors: Dusan Bosnjakovic

Keywords: Psychometrics, Multi-agent systems, Safety and bias

Source: Open primary source (opens in a new tab)

1
Authors
8
Findings
26
Limitations
5
Evidence

Editorial summary

English

The preprint proposes auditing provider-level alignment signatures through 2-4 sentence cloze vignettes. Each target blank has five options mapped in advance to an ordinal 1-5 scale and is hidden among semantically unrelated distractors. The manuscript says that multiple LLMs generate candidates, independent LLM judges retain items with a mean score of at least 4/5, and SHA-256(global_seed:item_id) determines ordering. It then groups nine models from OpenAI, Google, Anthropic, and xAI and lists a MixedLM with provider and item random effects, ICC, Kruskal-Wallis, Friedman, and post-hoc comparisons. It does not publish the nine identifiers, item count or texts, prompts, judges, calls, repetitions, dates, parameters, seeds, permutations, or sample sizes.

According to the reported results, Gemini has higher means on three forms of sycophancy, overconfidence, false balance, and economic-inequality valence; Claude scores lower on instrumentalization of humans, and GPT on false balance. Provider ICCs in Table 1 are small, ranging from 0.005 to 0.040. Table 2 marks eight of nine dimensions significant and one nonsignificant, although the prose and conclusion say seven of nine and another section names two nonsignificant dimensions; Conflict De-escalation, one of those two, is absent from the table. Normative descriptions such as dignity-centered Claude or moralizing Gemini follow from the author's 1-5 keys and are not validated against human judgment or an external criterion.

The statistical evidence does not support the stronger interpretation. An incompletely specified linear model is applied to ordinal responses, and no fitted IRT model, reliability, validity, or invariance evidence is provided to justify the psychometric framing. Provider has only four levels and is confounded with family, version, training, and alignment. The paper omits the model equation, fixed effects, blocking structure, estimator, variance-component test, statistics, degrees of freedom, intervals, and multiplicity correction; several p-values are printed as zero. Pole reversal would be algebraically guaranteed if the same responses are merely rescored, and the example is arithmetically inconsistent: 6-1.25 equals 4.75, not 4.65. Removing distractors changes the prompt but does not directly measure evaluation awareness.

No data, code, repository, supplement, or statistical output is available, and the arXiv source contains no additional artifacts. The bibliography contains many incorrect titles or metadata fields and three records for which no exact primary-source match could be located. The study also runs no multi-agent chains, recursive judges, mixed-provider systems, or longitudinal measurements. It therefore does not establish persistence across generations, causal lab policies, recursive amplification, or mitigation through provider diversity. Its defensible contribution is an exploratory vignette format and unreproduced reported differences in an undocumented sample, not established durable signatures or compounding risk.

Español

El preprint propone auditar supuestas firmas de alineamiento por proveedor mediante viñetas cloze de 2-4 oraciones. Cada blanco objetivo tiene cinco opciones preasignadas a una escala ordinal 1-5 y se oculta entre distractores semánticamente no relacionados. El manuscrito dice que varios LLM generan candidatos, jueces LLM independientes retienen los que alcanzan una media mínima de 4/5 y SHA-256(global_seed:item_id) determina el orden. Después agrupa nueve modelos de OpenAI, Google, Anthropic y xAI, y enumera MixedLM con efectos aleatorios de proveedor e ítem, ICC, Kruskal-Wallis, Friedman y comparaciones post-hoc. Sin embargo, no publica los nueve identificadores, el número o texto de los ítems, prompts, jueces, llamadas, repeticiones, fechas, parámetros, semillas, permutaciones ni tamaños muestrales.

Según los resultados reportados, Gemini obtiene medias más altas en tres formas de sycophancy, sobreconfianza, falso equilibrio y valencia ante desigualdad; Claude puntúa más bajo en instrumentalización humana, y GPT en falso equilibrio. Los ICC de proveedor de la Tabla 1 son pequeños, entre 0,005 y 0,040. La Tabla 2 marca ocho dimensiones significativas de nueve y una no significativa, aunque el texto y la conclusión afirman siete de nueve y otra sección dice que hay dos no significativas; Conflict De-escalation, una de ellas, ni siquiera aparece en esa tabla. Los efectos normativos -por ejemplo, llamar a Claude centrado en la dignidad o a Gemini moralizante- dependen de las claves 1-5 definidas por el autor y no se contrastan con humanos ni con un criterio externo.

La evidencia estadística no permite sostener la interpretación fuerte. Se aplica un modelo lineal sin especificación completa a respuestas ordinales y no se presenta el IRT, la fiabilidad, la validez o la invariancia que justificarían llamarlo marco psicométrico. Proveedor tiene solo cuatro niveles y queda confundido con familia, versión, entrenamiento y alineamiento. Faltan ecuación, efectos fijos, estructura de bloques, estimador, test de componentes, estadísticos, grados de libertad, intervalos y corrección por comparaciones múltiples; varios p se imprimen como 0. La inversión de polos sería algebraicamente obligada si solo se recodifica la misma respuesta y el ejemplo contiene un error: 6-1,25 es 4,75, no 4,65. Quitar distractores cambia el prompt, pero no mide directamente conciencia de evaluación.

No hay datos, código, repositorio, suplemento ni salida estadística, y el fuente arXiv no contiene artefactos adicionales. La bibliografía incluye numerosos títulos o metadatos incorrectos y tres registros sin coincidencia primaria exacta localizable. El estudio tampoco ejecuta cadenas multiagente, jueces recursivos, sistemas mixtos entre proveedores ni medidas longitudinales: por ello no demuestra persistencia entre generaciones, causalidad de políticas del laboratorio, amplificación recursiva ni que diversificar proveedores la reduzca. Su aportación defendible es plantear un formato exploratorio de viñetas y reportar diferencias no reproducibles en una muestra no documentada; no establecer firmas duraderas ni riesgo compuesto.

Research question

Can ordinal vignettes with distractors reveal response differences that cluster by provider, and do those differences justify speaking of durable alignment signatures with risk of amplification in multiagent systems from the same lab?

Method

Exploratory preprint without public artifact. It describes cloze vignettes with five preclassified options 1-5 and hidden target blanks among distractors; unidentified LLM generators create items and unidentified LLM judges filter them with mean >=4/5. It claims to evaluate nine models from four providers and analyze responses with MixedLM with random effects provider and item_id, ICC, Kruskal-Wallis, Friedman and post-hoc. It adds recoding by pole inversion and comparison with/without distractors, but does not specify the sample, the complete statistical model or the materials needed to reproduce them.

Sample: The manuscript claims to include nine models from four providers, but does not publish the complete list, the number of items, calls or repetitions, the observations per model/provider/dimension, data loss or the date of collection. Consequently, the sample size and the unit of analysis cannot be determined.

Findings

  • The paper reports provider ICC between 0.005 and 0.040 for six dimensions, equivalent to only 0.5%-4.0% of the variance under its unpublished specification.
  • It reports Gemini with higher authority sycophancy (1.93), epistemic (1.56) and emotional (1.72) than several competitors.
  • It reports GPT with lower false balance (1.32), Claude with lower instrumentalization (2.32) and Gemini with higher moral valence toward inequality (4.03).
  • Table 2 marks eight of nine rows significant, not seven of nine as the results section and the conclusion claim.
  • The text presents two non-significant dimensions, but Conflict De-escalation does not appear in the omnibus table and Overconfidence appears in prose but not in the statistical tables.
  • The pole inversion reports an approximate original mean of 1.25 and an inverted mean of 4.65, although the declared transformation would produce 4.75.
  • Upon removing distractors, the paper claims that significant differences go from 8 to 18 and H from 27.692 to 45.735, but provides no data and does not define the dimension, sample, degrees of freedom or complete tests.
  • None of the numerical results can be recomputed with the available public materials.

Limitations

  • The complete list of nine models is not published, nor immutable snapshots, dates or generation parameters.
  • The number of items, responses, calls, replicas, lost observations or cell sizes is not reported.
  • Vignettes, options, distractors, keys 1-5, prompts, dimensions or item assignments are not published.
  • LLM generators and judges are not identified; ratings, agreement, exclusions and human validation are missing.
  • The number and names of the dimensions change between introduction, results, tables and conclusion.
  • Ordinal responses 1-5 are analyzed using an unspecified linear model, without justifying continuity or normality.
  • IRT, Thurstonian IRT or MUPP invoked in the motivation is neither implemented nor reported.
  • There is no reliability, test information, dimensionality, construct validity, invariance or DIF.
  • Provider has only four levels and is confounded with model, family, version, training and alignment policy.
  • No equation, fixed effects, nesting/crossing, estimator, convergence, residuals or treatment of missingness is given.
  • The variance components sum to exactly one in each row, but their normalization is not explained.
  • Random effect asterisks do not include p, interval or method for testing the provider component.
  • Friedman requires a block or paired structure that the paper does not define; Kruskal-Wallis assumes independent groups.
  • Statistics, degrees of freedom, intervals and effect sizes for the omnibus tests are omitted.
  • Several p values are printed as 0.0/0.00 and no correction for multiplicity is declared.
  • Post-hoc comparisons do not specify test, exact p, effects, intervals or correction.
  • Pole inversion may be an algebraic identity and contains an arithmetic discrepancy.
  • The comparison with/without distractors alters the context and does not directly measure evaluation awareness.
  • There are no longitudinal collections, frozen versions or temporal replicas that test durability.
  • No multiagent system, recursive judge or chain across providers is executed.
  • Normative labels of labs depend on the author's scales without human or external criteria.
  • No data, code, repository, supplement, preregistration, promised hashes or statistical output are published.
  • The arXiv source does not include additional artifacts and the license is not an open reuse license such as CC BY.
  • The bibliography contains incorrect titles, authors, years or venues and three entries without an exact primary match that can be located.
  • There is no limitations, ethics, funding, conflicts, AI assistance or materials availability section.
  • The lab profile table on page 8 overflows the margin and leaves text visually cut off.

What the study does not establish

  • It does not establish durable or persistent alignment signatures across model generations.
  • It does not causally attribute differences to lab policies or to the alignment process.
  • It does not demonstrate personality, mind, internal values or ideology of the models or providers.
  • It does not demonstrate that provider is a stronger predictor than model, version or item.
  • It does not validate an ordinal psychometric instrument or a latent variable comparable across models.
  • It does not demonstrate that distractors hide intent, prevent gaming or simulate real deployment.
  • It does not demonstrate amplification, echo chambers or compounding bias in multiagent systems.
  • It does not demonstrate that mixing providers reduces bias or increases safety.
  • It does not allow verifying significance, sample sizes, robustness or generalization of the results.
  • It does not justify treating the normative labels assigned to each lab as general empirical facts.
  • It does not establish validity outside the prompts, scales, language and undocumented models of the study.

Traceability

Scope: Full text

Version: arXiv:2602.17127v1, submitted 2026-02-19; arXiv nonexclusive distribution license 1.0

Consulted source: https://arxiv.org/abs/2602.17127

Review: Codex 13-page visual full-text, complete arXiv source, model/sample, ordinal-measurement, MixedLM/ICC, internal-consistency, citation, artifact and claim-boundary audit, 2026-07-18

Approval: Codex fidelity pass, 2026-07-18

English translation: approved, 2026-07-18

Models evaluated

  • Nueve modelos no enumerados completamente de OpenAI, Google, Anthropic y xAI
  • GPT-4, mencionado como contraste generacional sin identificador ni resultados completos
  • GPT-5, sin snapshot ni parámetros
  • Gemini 2.0 Flash, sin snapshot ni parámetros
  • Gemma-3-27b-it, sin snapshot ni parámetros
  • Claude, familia no desglosada en modelos exactos
  • Grok, familia no desglosada en modelos exactos

Instruments and metrics

  • Viñetas cloze de 2-4 oraciones con cinco opciones ordinales 1-5
  • Blanco objetivo entre distractores semánticamente ortogonales
  • Filtrado de ítems por jueces LLM con media declarada >=4/5
  • Orden determinista declarado mediante SHA-256(global_seed:item_id)
  • Pole reversal de la escala de falso equilibrio
  • Mixed Linear Model con proveedor e item_id como efectos aleatorios
  • ICC de proveedor
  • Kruskal-Wallis, Friedman y comparaciones post-hoc no especificadas

Data used

  • Matriz de respuestas de nueve modelos descrita pero no publicada
  • Banco de viñetas, opciones y distractores descrito pero no publicado
  • Resultados de jueces LLM e informes estadísticos no publicados

Evidence and location

  • Design, results, tables, discussion, conclusion, bibliography and visual defect: arXiv:2602.17127v1, all 13/13 PDF pages rendered and individually inspected
  • Version, date, authorship, category, links and license: Official arXiv abstract and Atom metadata inspected 2026-07-18
  • Absence of appendices, data, code and hidden materials: Complete official arXiv v1 source archive inspected; paper.tex sha256 78ac46fa05ea89fd662fcfd35a3d2dd6b0ecf453727b6e25328daa3db9632074
  • Correction of traceable references and detection of records without exact match: Official ACL Anthology, arXiv and Nature records cached under .cache/editorial-sources/article-400/secondary
  • Audit of sample, measurement, statistics, contradictions, citations, reproducibility and limits: reports/verification/article-400-lab-signature-missing-sample-ordinal-mixed-model-provider-confounding-citation-and-artifact-audit.json