Quantifying and Mitigating Socially Desirable Responding in LLMs: A Desirability-Matched Graded Forced-Choice Psychometric Study

Evaluation and psychometric validity2026arXivApproved editorial review

Authors: Kensuke Okada, Yui Furukawa, Kyosuke Bunji

Keywords: Personality, Persona conditioning, Psychometrics, Safety and bias

Source: Open primary source (opens in a new tab)

3
Authors
14
Findings
34
Limitations
6
Evidence

Editorial summary

English

The paper studies instructed socially desirable responding in LLM personality questionnaires and proposes a graded forced-choice (GFC) mitigation. Starting from 100 public-domain IPIP Big Five markers, GPT-5 and Gemini 2.5 Pro each provide 30 desirability ratings per item; after two voting items are excluded, optimization selects 30 cross-domain pairs containing 60 unique items with closely matched desirability. Fifty correlated synthetic Big Five vectors are discretized into stanines and verbalized through explicit adjective profiles. Nine models answer under HONEST and FAKE-GOOD instructions in both a 60-item Likert format and a 30-pair GFC format. Every item or pair is a separate API call that repeats the entire profile, rather than a continuous questionnaire session or spontaneously observed personality.

The released data contain 54,000 complete Likert responses and 26,999 of 27,000 expected GFC responses. Across the 45 model-by-trait cells, every desirability-direction-corrected Likert effect is positive, ranging from d=0.628 to 1.695 with mean 1.116. GFC sharply reduces that tendency: effects range from -0.917 to 0.666, with signed mean -0.089 and mean absolute effect 0.298. Honest profile recovery against the authors' continuous generated z target is higher for Likert (model means 0.866-0.947) than GFC (0.420-0.695). The defensible contribution is evidence that desirability matching and comparative judgment attenuate the average shift toward looking good, while losing information and not making each trait invariant.

The audit finds important limitations hidden by the main aggregation. Figure 4 averages the five signed trait effects before applying absolute practical thresholds: for GPT-5, A=0.462 and C=0.360 cancel against N=-0.388 and O=-0.873, yielding a -0.058 mean labeled practically negligible despite large profile changes. GPT-5 mini reaches -0.917 on N and Gemini 2.5 Pro -0.755 on O. The IRT models pool all nine models and both conditions under shared item parameters without testing measurement invariance or differential item functioning. The Likert GRM also retains 245 parameters above the declared Rhat 1.01 threshold, with maximum 1.0348 and minimum bulk ESS 115.55, although the manuscript only says convergence was monitored and the pipeline proceeds to posterior means. Tests, correlations and plotted intervals then treat those theta means as error-free observations rather than propagating posterior uncertainty.

The sole allegedly missing Claude Sonnet GFC answer is actually present in the published raw CSV and ends in the valid value 3; a stale derivative turns it into NA, the retry was never retrieved and the analysis removes all five theta values for that persona-condition. The OSF release is broad -R/Stan code, data, provider states, fitted models, diagnostics and figures- and its revised figures hash-match arXiv v2, but the project is unregistered and mutable, has no node license and does not freeze R, package or CmdStan versions. OpenAI and Gemini were called through provider aliases and defaults without released proof of the immutable snapshots suggested by the paper's table. The study supports GFC as a promising mitigation in this explicit synthetic design; it does not establish general absence of bias, full profile preservation, stable personality, spontaneous SDR or cross-model, cross-cultural and deployment-level psychometric validity.

Español

El artículo estudia la respuesta socialmente deseable inducida en cuestionarios de personalidad para LLM y propone mitigarla con un formato graded forced-choice (GFC). A partir de 100 marcadores Big Five de IPIP, GPT-5 y Gemini 2.5 Pro realizan 30 valoraciones de deseabilidad por ítem; tras excluir dos ítems sobre votar, una optimización selecciona 30 pares entre dominios y 60 ítems únicos con deseabilidad muy próxima. Cincuenta perfiles sintéticos se generan como vectores Big Five correlacionados, se discretizan en estaninos y se verbalizan mediante adjetivos explícitos. Nueve modelos responden bajo instrucciones HONEST y FAKE-GOOD, tanto a 60 ítems Likert como a 30 pares GFC. Cada ítem o par constituye una llamada independiente que vuelve a incluir el perfil completo; no es una sesión continua ni una personalidad observada espontáneamente.

Los datos liberados contienen 54.000 respuestas Likert completas y 26.999 de las 27.000 GFC previstas. En los 45 cruces modelo×rasgo, los efectos Likert corregidos hacia la dirección socialmente deseable son siempre positivos, entre d=0,628 y 1,695, con media 1,116. GFC reduce con fuerza esa tendencia: sus efectos van de -0,917 a 0,666, la media con signo es -0,089 y la media absoluta 0,298. La recuperación honesta del perfil, correlacionada con el z continuo generado por los autores, es mayor en Likert (medias por modelo 0,866-0,947) que en GFC (0,420-0,695). La contribución defendible es mostrar que el emparejamiento por deseabilidad y la comparación forzada atenúan el desplazamiento promedio en la dirección de quedar bien, aunque sacrifican información y no mantienen invariable cada rasgo.

La auditoría identifica límites importantes que la agregación principal oculta. La Figura 4 promedia los cinco efectos con signo antes de aplicar umbrales absolutos: en GPT-5, por ejemplo, A=0,462 y C=0,360 se cancelan con N=-0,388 y O=-0,873, produciendo una media de -0,058 que aparece como prácticamente despreciable pese a cambios grandes del perfil. GPT-5 mini alcanza -0,917 en N y Gemini 2.5 Pro -0,755 en O. Los modelos IRT agrupan los nueve modelos y ambas condiciones con parámetros de ítem compartidos, sin probar invariancia o funcionamiento diferencial. Además, el GRM Likert conserva 245 parámetros con Rhat>1,01 -máximo 1,0348 y ESS bulk mínimo 115,55-, aunque el manuscrito solo dice que se monitorizó la convergencia y el pipeline continúa con medias posteriores. Los tests, correlaciones e intervalos gráficos tratan esas medias theta como observaciones sin error y no propagan incertidumbre posterior.

La única respuesta GFC supuestamente ausente de Claude Sonnet está realmente en el CSV bruto publicado y termina en el valor válido 3; un derivado desfasado la convierte en NA, el reintento quedó sin recuperar y el análisis elimina los cinco theta de esa persona-condición. El OSF es amplio -código R/Stan, datos, estados, modelos ajustados, diagnósticos y figuras- y las figuras revisadas coinciden por hash con arXiv v2, pero el proyecto no está registrado ni congelado, carece de licencia de nodo y no fija versiones de R, paquetes o CmdStan. OpenAI y Gemini se llamaron mediante aliases y defaults del proveedor, sin acreditar los snapshots inmutables que sugiere la tabla del paper. El trabajo apoya GFC como mitigación prometedora en este diseño explícito y sintético; no demuestra ausencia general de sesgo, preservación completa del perfil, personalidad estable, SDR espontánea ni validez psicométrica transversal entre modelos, culturas o despliegues.

Research question

How much does an explicit instruction to present oneself favorably distort the estimated Big Five profiles in nine LLMs, and can a GFC inventory of pairs matched on desirability reduce that shift without losing the synthetic profile signal indicated in the prompt?

Method

100 IPIP items are rated on desirability using 30 replicas of GPT-5 and Gemini 2.5 Pro; two items are excluded and an integer optimization selects 30 pairs across domains, 60 unique items and small desirability differences. 50 correlated Big Five vectors are generated, transformed into stanines and explicit descriptions, and reused for nine models. The design crosses HONEST/FAKE-GOOD with a 60-response Likert format and a 30-comparison GFC format, sending each item/pair in an independent call. A GRM and an ordinal Thurstonian model, adjusted by format and grouping models/conditions, produce posterior thetas. Paired d_z are calculated per trait and model, t tests with Bonferroni correction and correlations between honest theta and the continuous z of the person.

Sample: The design uses 50 synthetic persons common to nine models, two conditions and two formats. 81,000 independent calls were expected: 54,000 Likert and 27,000 GFC. The 54,000 Likert and 26,999 GFC are published; one Claude Sonnet response was marked as absent in a derived file although it exists and is valid in the raw. Effects are calculated over 50 HONEST/FAKE-GOOD pairs per model and trait, except the five GFC cells of Claude Sonnet that remain with 49 pairs after completely excluding one person-condition.

Findings

  • The 45 direction-corrected Likert effects are positive, between 0.628 and 1.695, with a mean of 1.116.
  • The GFC effects range from -0.917 to 0.666; their signed mean is -0.089 and their absolute mean 0.298.
  • The paper reports the 45 Likert effects and only 4 of 45 GFC as significant at 1%.
  • The mean correlation per model with continuous z is between 0.866 and 0.947 in Likert and 0.420 and 0.695 in GFC.
  • The correlation with the actually verbalized stanine systematically exceeds the correlation with z by 0.015-0.035.
  • GFC clearly attenuates the average shift toward the socially desirable direction, but does not eliminate large changes by trait.
  • GPT-5 GFC has a signed mean of -0.058, an absolute mean of 0.446 and a maximum absolute effect of 0.873.
  • GPT-5 mini GFC has a signed mean of -0.183, an absolute mean of 0.514 and a maximum absolute effect of 0.917.
  • Gemini 2.5 Pro GFC has a signed mean of -0.072, an absolute mean of 0.432 and a maximum absolute effect of 0.755.
  • The 30 published pairs reach a maximum desirability difference of 0.18 and an approximate mean of 0.03 according to the released solution.
  • The GRM has 245 parameters with Rhat>1.01, maximum 1.034786; the GFC model has no parameters above the threshold.
  • The single response considered absent is present in raw_anthropic_main.csv and can be parsed as 3.
  • The revised OSF figures match by hash with those included in the arXiv v2 source.
  • The central evidence describes differential compliance with an explicit instruction, not spontaneous personality or desirability.

Limitations

  • FAKE-GOOD explicitly orders giving a good impression; it does not test spontaneous SDR under implicit evaluation.
  • Each item or pair is an independent call and repeats the complete person, not a continuous questionnaire session.
  • The profiles explicitly list the five traits; recovery measures prompt following, not established personality.
  • The assumed ground truth is a synthetic construction by the authors, not observed human behavior or trait.
  • The model receives verbalized stanines, but the main recovery is correlated with the continuous z not shown.
  • Likert obtains 60 responses and GFC 30 comparisons; neither the number of responses nor the test information are equated.
  • The IRT models share item parameters across nine models and two conditions without a test of invariance.
  • Differential item functioning is not evaluated by provider, capacity, model or condition.
  • HONEST and FAKE-GOOD are treated as separate units in IRT; dependence by person only reappears when calculating effects.
  • Posterior analyses use theta means and omit their posterior uncertainty.
  • The intervals in Figure 2 are ordinary standard errors of theta means, not full posterior intervals.
  • The GRM uses only 200 warmup iterations and retains 245 parameters above the declared Rhat.
  • The manuscript does not quantify or discuss those 245 convergence failures.
  • Figure 4 averages signed effects and allows positive and negative changes to cancel.
  • Applying |d| after averaging signs does not guarantee that each trait has a negligible change.
  • The green/yellow/red thresholds are not validated as decision criteria for synthetic LLM profiles.
  • No equivalence tests are run to support the practically negligible label.
  • The human benchmark comes from another meta-analysis and non-equated instruments and is only descriptive.
  • The human validation of desirability uses different adjectives, not the 98 IPIP statements that are paired.
  • GPT and Gemini participate in the construction of pairs and then their families are evaluated with those same pairs.
  • A valid raw response is lost due to drift between raw and parsed and causes the exclusion of five thetas.
  • The retry state of that response remained in_progress and expired without a recovered result.
  • The aliases gpt-5 and Gemini do not freeze a serving revision nor allow crediting all the snapshots in the table.
  • Temperature, top-p or generation seed are not fixed; provider defaults are used.
  • Commercial collection may vary over time and costs 81,000 calls for a complete replication.
  • The OSF is not registered and may be modified after review.
  • The OSF node does not declare a license and the README does not separately license code and data.
  • There is no lockfile, package versions, sessionInfo, CmdStan version, container or system capture.
  • The sample is 50 synthetic profiles and nine models; uncertainty about generalization across models is limited.
  • Only Big Five is studied in English with a specific selection of 60 items.
  • The instrument is not validated with humans responding to the same pairs under the same scoring model.
  • No other languages, cultures, inventories, narrative profiles, long conversations or real deployments are tested.
  • Interpreting effects contrary to desirability as lower bias may hide another form of distortion.
  • The association with safety, fairness or value surveys is proposed as an application, but is not empirically tested.

What the study does not establish

  • It does not establish that models possess stable, internal or human Big Five personality.
  • It does not establish spontaneous SDR or autonomous detection of an evaluative context.
  • It does not establish that GFC eliminates bias or preserves all traits of the profile.
  • It does not justify that every point in the green zone of Figure 4 is a recommended option.
  • It does not establish measurement invariance or absence of DIF across models and conditions.
  • It does not demonstrate satisfactory convergence of the GRM with the declared threshold.
  • It does not incorporate posterior uncertainty into effects, tests, correlations or intervals.
  • It does not demonstrate that continuous z is the visible ground truth to the model.
  • It does not establish psychometric equivalence with humans or with the meta-analysis used as a reference.
  • It does not test validity for other traits, inventories, languages, cultures or interfaces.
  • It does not test utility or safety in selection, mental health, education, fairness or high-impact decisions.
  • It does not unambiguously freeze the OpenAI/Gemini models, dependencies or generation configuration.
  • It does not allow an exact, free and deterministic replication of the 81,000 calls.
  • It does not convert compliance with explicit profiles into evidence of persistent behavior outside the prompt.

Traceability

Scope: Full text

Version: arXiv:2602.17262v2, revised 2026-04-28, CC BY 4.0; full OSF project 2e6ny inspected

Consulted source: https://arxiv.org/abs/2602.17262

Review: Codex 19-page visual full-text, arXiv source, complete OSF archive, raw-call reconciliation, R/Stan, IRT, convergence, measurement-invariance, signed-cancellation, artifact and claim audit, 2026-07-18

Approval: Codex fidelity pass, 2026-07-18

English translation: approved, 2026-07-18

Models evaluated

  • gpt-5, alias del proveedor; la resolución al snapshot 2025-08-07 no aparece en los datos brutos
  • gpt-5-mini, alias del proveedor; la resolución al snapshot 2025-08-07 no aparece en los datos brutos
  • gpt-5-nano, alias del proveedor; la resolución al snapshot 2025-08-07 no aparece en los datos brutos
  • gemini-2.5-pro
  • gemini-2.5-flash
  • gemini-2.5-flash-lite
  • claude-opus-4-5-20251101
  • claude-sonnet-4-5-20250929
  • claude-haiku-4-5-20251001

Instruments and metrics

  • 100 marcadores Big Five de IPIP, reducidos a 98 tras excluir dos ítems sobre votar
  • Valoración LLM de deseabilidad social en escala 1-9
  • Optimización mixta entera en dos etapas para seleccionar pares
  • Inventario Likert de 60 ítems y siete puntos
  • Inventario graded forced-choice de 30 pares y siete puntos
  • Cincuenta perfiles Big Five Gaussianos transformados en estaninos y frases adjetivales
  • Instrucciones explícitas HONEST y FAKE-GOOD
  • Graded Response Model bayesiano para Likert
  • Modelo Thurstoniano ordinal bayesiano para GFC
  • Cohen d_z corregido por dirección, tests t pareados y Bonferroni sobre 90 celdas
  • Correlación de Pearson entre theta honesto y z/estanino de la persona
  • Benchmark humano descriptivo importado de la metaanálisis de Speer et al. (2023)

Data used

  • 54.000 respuestas Likert publicadas: 6.000 por cada uno de nueve modelos
  • 26.999 respuestas GFC publicadas: 3.000 por modelo salvo 2.999 para Claude Sonnet
  • persona_theta_long.csv: 8.995 medias theta de modelo, formato, condición, persona y rasgo
  • rhat_grm_over_1.01.csv: 245 parámetros GRM por encima del umbral declarado
  • rhat_fc_over_1.01.csv: ningún parámetro GFC por encima de 1,01
  • Resultados de 6.000 valoraciones de deseabilidad realizadas por GPT-5 y Gemini 2.5 Pro
  • Solución publicada de 30 pares y metadatos de 60 ítems
  • Estados y CSV brutos de 81.000 solicitudes esperadas a tres proveedores
  • Archivo OSF 2e6ny de 176 ficheros, sha256 af406ddae238b3984bede3a509a562e0e24d4a599ef07fdf36fd3df318ac8f87
  • Fuentes TeX y figuras de arXiv:2602.17262v2

Evidence and location

  • Design, theory, results, figures, appendices, limitations, ethics and AI assistance: arXiv:2602.17262v2, all 19/19 PDF pages rendered and individually inspected
  • R/Stan code, data, provider states, settings, diagnostics, figures and README: Official public OSF project 2e6ny, complete 176-file archive downloaded and hash-validated
  • Version, dates, authorship, license and correspondence of revised figures: Official arXiv abstract, Atom metadata and v2 TeX/source archive inspected 2026-07-18
  • Reproduction of d_z, recovery with z/stanines and signed versus absolute cancellation: Independent recomputation from OSF persona_theta_long.csv sha256 1e0e77bb881961818d3e6df5a890a26955c7b5b526cba2ed8c0aa06eef9cc4b4
  • Recoverable Claude Sonnet response and drift between raw, parsed, best files and refit state: OSF raw_anthropic_main.csv, raw_anthropic_main_parsed.csv, fc7_calls_raw.csv and state_anthropic_refit01.json inspected for req_id fc7_s49_honest_b29
  • Audit of convergence, invariance, uncertainty, aggregation, artifact, license and limits: reports/verification/article-399-sdr-gfc-irt-measurement-invariance-signed-cancellation-convergence-artifact-and-claim-audit.json