Synthetic Personality Review

Persona Non Grata: LLM Persona-Driven Generations in MCQA are Unstable in Distinct Dimensions

Personas, identity, and agents2026arXivApproved editorial review

Authors: César Guerra-Solano, Xiang Lorraine Li

Keywords: Persona conditioning, MCQA, Prompt sensitivity, Evaluation robustness, Sampling temperature, Rank instability, Question-level consistency, Demographic persona prompting

Source: Open primary source (opens in a new tab)

2
Authors
8
Findings
10
Limitations
6
Evidence

Editorial summary

English

This preprint studies how much persona-conditioned MCQA conclusions change when persona prompts, task prompts, and temperature vary. Although one methods sentence says four models, the tables and analyses evaluate five: Llama-3.2-1B-Instruct, Llama-3.1-8B-Instruct, and Qwen2.5-Instruct at 1.5B, 7B, and 14B. Each model answers a 5,013-question composite of MMLU, Social IQa, and NormAd-Eti under 41 labels and 48 configurations: four persona-prompt formats, four task-prompt formats, and temperatures 0, 0.5, and 1. The 41 labels consist of 39 roles or identities plus two controls, `a human` and `NO PERSONA`, so aggregates described as persona results also include controls. The design implies 1,968 full evaluations and 9,865,584 question outputs per model. The authors propose three metrics. IA is the mean absolute accuracy difference over pairs of configurations and is a transparent descriptive sensitivity measure. IO summarizes weighted variation in label ranks, but its setting weights and final scale factor depend on between-label accuracy dispersion, so it is not a pure rank-change measure. IQ is one minus the intersection of correct-question sets across every configuration divided by their union. This definition mechanically depends on the number of configurations: with 48 conditions, requiring a question to be correct in all of them shrinks the intersection even when accuracy probability is stable. The paper's own sensitivity table demonstrates the effect. Moving from 12 to 48 configurations increases IQ from 61.083 to 78.293 for Llama-8B and from 32.061 to 52.828 for Qwen-14B, changes inconsistent with describing the metric as only slightly variable. Its .948-.990 correlation with Fleiss' kappa uses the same binary correctness matrix and is not independent construct validation. Descriptively, smaller models are generally more sensitive. Qwen-7B has the lowest IA and IO, while Qwen-14B improves IQ but is slightly worse than 7B on IA and IO. Task-prompt format is the strongest reported factor, but its levels switch between option-only answering and explanation generation and also change whether the model is reminded of its persona; this is more than surface wording. Math and commonsense/social reasoning show greater sensitivity, and configurations can change the best and worst labels, a useful warning against single-configuration persona rankings. Four settings designated stable at temperature 0 outperform four temperature-1 settings in accuracy for four of five models, but selection and evaluation use the same data, while the metrics derive from accuracy or its ranks. The comparison is exploratory, not causal or held out. Each stochastic cell is observed once, so prompt or temperature effects cannot be separated from random realization variance. No public outputs, composite dataset, labeling code, parser, metric code, seeds, exact model revisions, or analysis scripts were located. The defensible conclusion is that conclusions from this persona-MCQA protocol can materially depend on evaluation configuration, not that all three metrics are validated, setting-invariant dimensions or that demographic labels reflect human differences.

Español

Este preprint estudia cuánto cambian las conclusiones de evaluaciones MCQA condicionadas por una persona cuando varían el prompt de persona, el prompt de tarea y la temperatura. Aunque una frase de métodos dice cuatro modelos, las tablas y análisis evalúan cinco: Llama-3.2-1B-Instruct, Llama-3.1-8B-Instruct y Qwen2.5-Instruct de 1,5B, 7B y 14B. Cada modelo responde las 5.013 preguntas de un compuesto de MMLU, Social IQa y NormAd-Eti bajo 41 etiquetas y 48 configuraciones: cuatro formatos de prompt de persona, cuatro de tarea y temperaturas 0, 0,5 y 1. Las 41 etiquetas incluyen 39 roles o identidades y dos controles, `a human` y `NO PERSONA`; por tanto, los agregados denominados persona también incluyen controles. El diseño implica 1.968 evaluaciones completas y 9.865.584 respuestas a preguntas por modelo. Los autores proponen tres métricas. IA es la diferencia absoluta media de accuracy entre pares de configuraciones y ofrece una medida descriptiva clara. IO resume la desviación ponderada de los rankings de etiquetas, pero sus pesos y su factor final dependen de la dispersión de accuracy entre personas; no es una medida pura de cambio de rango. IQ calcula uno menos la intersección de preguntas correctas en todas las configuraciones dividida por su unión. Esta definición depende mecánicamente del número de configuraciones: con 48 condiciones, exigir que una pregunta sea correcta en todas hace que la intersección se reduzca aunque la probabilidad de acierto sea estable. La propia tabla de sensibilidad lo muestra: al pasar de 12 a 48 configuraciones, IQ sube de 61,083 a 78,293 para Llama-8B y de 32,061 a 52,828 para Qwen-14B, cambios incompatibles con describir la métrica como apenas variable. Su correlación de .948–.990 con Fleiss kappa usa la misma matriz binaria de aciertos y no valida de forma independiente el constructo. Descriptivamente, los modelos pequeños son por lo general más sensibles; Qwen-7B presenta los menores IA e IO, mientras Qwen-14B mejora IQ pero empeora ligeramente IA e IO respecto a 7B. El formato de tarea es el factor más influyente, pero sus niveles cambian entre responder solo una opción y generar una explicación, además de recordar o no la persona: no es únicamente una variación superficial de redacción. Matemáticas y razonamiento social/commonsense muestran más sensibilidad y las configuraciones pueden cambiar la mejor y peor etiqueta, un resultado útil para advertir contra rankings de personas basados en una sola configuración. La comparación entre cuatro condiciones llamadas estables a temperatura 0 y cuatro llamadas inestables a temperatura 1 favorece la accuracy en cuatro de cinco modelos, pero selección y evaluación se hacen con los mismos datos y las métricas derivan de accuracy o sus rankings; es evidencia exploratoria, no causal ni held-out. Cada celda estocástica se observa una sola vez, de modo que no se separa el efecto de temperatura o prompt de la variación de la realización aleatoria. Tampoco se publican salidas, dataset compuesto, código de etiquetado, parser, métricas, semillas, revisiones exactas de modelos o scripts de análisis. El hallazgo defendible es que las conclusiones de este protocolo persona-MCQA pueden depender materialmente de la configuración; no que las tres métricas sean dimensiones validadas e invariantes, ni que las etiquetas demográficas reflejen diferencias humanas.

Research question

How much do accuracy, the ranking among persona labels, and the set of correct questions of an MCQA evaluation change when varying the persona prompt format, the task prompt format, and the temperature?

Method

Descriptive factorial study with five open-weight LLMs from two families. It crosses 41 labels (39 roles/identities and two controls) with 48 configurations of four persona prompts, four task prompts, and three temperatures. Each combination is applied once to 5,013 questions from ten domains sourced from MMLU, Social IQa, and NormAd-Eti. It computes absolute differences in accuracy, weighted variation of rankings, and multiset Jaccard of correct questions; it stratifies by model, hyperparameter, domain, and persona group, compares configurations designated stable/unstable, and adds a roberta-base classifier to predict the best labels.

Sample: Five instructed models from Llama and Qwen; 41 labels grouped by gender, sexuality, race/ethnicity, age, disability, religion, politics, occupation, and two controls; 48 configurations; 5,013 questions. There are 1,968 complete evaluations and 9,865,584 outputs per model, 49,327,920 implicit outputs in total.

Findings

  • Conclusions about accuracy and ranking among labels change with configuration; different conditions produce different best and worst labels, so a single configuration does not support a robust ranking of personas.
  • Smaller models tend to show greater sensitivity, but Qwen-14B has somewhat worse IA and IO than Qwen-7B; five checkpoints from two families do not establish a general scaling law.
  • The task format shows the strongest association, but it also changes between short answer and explanation and modifies persona salience; it does not isolate a merely formal effect of the prompt.
  • Mathematics and commonsense/social reasoning show greater descriptive sensitivity; attributing this to the composition of the training corpus is an untested hypothesis.
  • IA transparently summarizes accuracy changes; IO mixes ranking movement and dispersion among labels; IQ strongly depends on the number of configurations because it uses the intersection of all of them.
  • The published sensitivity of IQ between 12 and 48 configurations reaches 17.21 points in Llama-8B and 20.77 in Qwen-14B, contradicting the description of light variation.
  • Conditions called stable at temperature 0 have higher accuracy in four of five models, with Qwen-14B as the exception; the comparison is exploratory and selected on the same data.
  • Reported parsing errors are small relative to total failures and do not appear to explain the aggregate differences, but there are no public outputs or parser to verify this.

Limitations

  • Each model-label-question-configuration cell has a single output despite `do_sample=True`; repetitions and seeds are missing to separate stochastic variation from hyperparameter effects.
  • IQ requires simultaneous correctness across all configurations and by construction increases as conditions are added; it is not comparable without fixing the number and composition of configurations.
  • The correlation between IQ and Fleiss kappa uses the same binary matrix and does not constitute independent validation or demonstrate three separable dimensions.
  • Outputs, composite dataset, automatic labeling of Social IQa, runner, parser, metric code, analysis scripts, seeds, exact revisions, or environment are not published.
  • There are no uncertainty intervals or a model for the dependence generated by reusing the same questions across 41 labels and 48 configurations.
  • The stability-accuracy comparison selects and evaluates conditions on the same sample and uses metrics defined from accuracy or its rankings.
  • The 41 supposed personas include two controls; the demographic labels are coarse, not intersectional, and not validated with the represented communities.
  • The hypothesis that domain sensitivity stems from corpus deficiencies is not tested, and benchmark contamination is not audited either.
  • The auxiliary classifier does not document in a self-contained manner the split, label construction, seeds, hyperparameters, uncertainty, or numerical results.
  • The methods text states four models although the study analyzes five.

What the study does not establish

  • It does not establish that all persona-guided generation is unstable, nor does it generalize beyond five instructed models, two families, MCQA in English, and these prompts.
  • It does not isolate instability caused by introducing a persona, because it studies sensitivity within the paradigm and aggregates 39 labels with two controls.
  • It does not validate IA, IO, and IQ as three independent constructs, invariant to the number of conditions, or comparable across designs.
  • It does not causally demonstrate that task format or temperature produce the observed effects, nor does it separate prompt semantics from sampling randomness.
  • It does not demonstrate that choosing the conditions called stable causally improves accuracy on new data or models.
  • It does not allow interpreting differences among demographic labels as capabilities, personalities, or individual variation of human groups.
  • It does not support the causal explanation based on the amount of training data per domain.
  • It does not offer an independent computational reproduction nor allow auditing the parser, the nearly 49.3 million implicit results, or the auxiliary classifier.

Traceability

Scope: Full text

Version: arXiv:2607.00937v1; preprint without an identified accepted venue

Consulted source: https://arxiv.org/pdf/2607.00937

Review: Codex 23-page full-text visual, TeX, publication, metric, statistical, artifact, reproducibility and claim audit, 2026-07-16

Approval: Codex fidelity pass, 2026-07-16

English translation: approved, 2026-07-18

Models evaluated

  • Llama-3.2-1B-Instruct
  • Llama-3.1-8B-Instruct
  • Qwen2.5-1.5B-Instruct
  • Qwen2.5-7B-Instruct
  • Qwen2.5-14B-Instruct
  • roberta-base auxiliary best-persona classifiers

Instruments and metrics

  • Performance instability IA: mean absolute pairwise accuracy difference
  • Outcome instability IO: weighted standard deviation of persona ranks scaled by mean between-persona spread
  • Question correctness instability IQ: one minus all-setting Jaccard intersection over union
  • Fleiss' kappa comparison
  • Four persona prompt formats and four task prompt formats
  • Parsing-error rate

Data used

  • MMLU
  • Social IQa
  • NormAd-Eti
  • Unreleased 5,013-question composite grouped into ten domains

Evidence and location

  • Preprint status, authors, date, and scope: Official arXiv record 2607.00937v1 and 23-page PDF, checked 2026-07-16
  • Models, 41 labels, 5,013 questions, and 48 configurations: arXiv v1, Sections 4-5 and Appendices A-D
  • Definitions of IA, IO, and IQ: arXiv v1, Section 3, Instability Metrics
  • Results by model, hyperparameter, domain, accuracy, and ranking changes: arXiv v1, Sections 5-6 and Tables 1 and 4
  • Dependence of IQ on the number of configurations, correlations, parsing, and classifier: arXiv v1, Appendix E and Tables 8, 10, and 11
  • Audit of metrics, artifacts, reproducibility, and claim boundaries: reports/verification/article-285-arxiv-persona-mcqa-instability-metric-setting-count-replication-artifact-and-claim-audit.json