Fairness Testing of Large Language Models in Role-Playing

Applications, bias, and safety2026ACMApproved editorial review

Authors: Xinyue Li, Zhenpeng Chen, Jie M. Zhang, Ying Xiao, Tianlin Li, Weisong Sun, Yang Liu, Yiling Lou, Xuanzhe Liu

Keywords: BiasLens, Role-playing prompts, Fairness testing, Social bias, Adversarial test generation, LLM-as-judge, Demographic roles, Test oracle validation

Source: Open primary source (opens in a new tab)

9
Authors
10
Findings
14
Limitations
7
Evidence

Editorial summary

English

This FSE 2026 paper presents BiasLens, an adversarial test generator for exposing socially biased responses when an LLM is assigned a role. GPT-4o generates 50 roles for each of 11 categories and 20 Yes/No, 20 Choice, and 20 Why questions per role. After removing 136 questions containing Group A/B placeholders, the effective set contains 32,864 questions rather than 33,000. Ten models answer every question three times in independent conversations, producing 985,920 raw responses. For each model-question combination, BiasLens uses a two-of-three majority: Yes and Choice use predefined rules, while Why uses three independent GPT-4o-mini votes per response followed by a majority across the three responses. The reported 107,580 cases are not individual responses. They are model-question combinations classified as biased among 328,640 possible combinations, or 32.73% within this adversarial test. Per-model counts range from 7,579 for GLM-4.5 to 16,963 for Llama-3-8B. Removing the role prefix from the same questions lowers the published total from 107,580 to 81,497, or 24.80% of cases and an absolute reduction of 7.94 percentage points. The paper reports a mean model-level relative reduction of 23.8%. This paired contrast is the most useful result: for questions expressly designed to trigger stereotypes, adding these roles consistently increases detections across all ten systems. It is not an estimate of prevalence in ordinary use. Roles are generated by requesting groups likely to discriminate and questions are written to elicit bias; categories such as character, social, and victim are not homogeneous demographic attributes either. Oracle validity is uneven. In human samples, the universal No rule for Yes/No agrees in 94.6% of cases and the final-option Choice rule in 94.4%; items with an invalid oracle are not removed from the headline total. For Why, the judge agrees with humans on only 80.7% of 384 sampled pairs, and Why questions contribute 54,538 detections. The paper highlights a 2.6% missed-bias rate but does not release the confusion matrix needed to explain the total 19.3% disagreement. The repository audit finds an additional defect: the Choice parser treats any text containing an uppercase C as neutral. Answers such as A) Children and A. College graduates are therefore read as option C. A conservative first-option reparse changes 6,383 role-conditioned model-question cases. For Qwen1.5-110B, the count changes from 7,754 to approximately 13,278. This audit reparse is not a substitute for new human annotation, but it invalidates exact parser-based comparisons and rankings. The aggregate role-removal effect remains descriptively large because the same defect affects both conditions. The paper also applies a two-proportion z-test as if the samples were independent even though each question is observed with and without the role; a paired analysis is required. The artifact releases nearly two million responses, questions, and result files, which is valuable, but it is not self-consistent. Current GLM raw responses regenerate 22 fewer detections with roles and 19 fewer without roles than the saved tables. Human annotations, the role generator, and the code that creates the final judge label are absent; several scripts target nonexistent directories, and there are no pinned versions, tests, CI, or LICENSE file. The defensible conclusion is that specific role prefixes amplify bias detections in an adversarial test bank, not that 32.73% of real interactions are biased or that the published ranking stably measures general model fairness.

Español

Este artículo publicado en FSE 2026 presenta BiasLens, un generador de pruebas adversariales para descubrir respuestas socialmente sesgadas cuando un LLM recibe un rol. GPT-4o genera 50 roles para cada una de 11 categorías y 20 preguntas Yes/No, 20 Choice y 20 Why por rol. Tras retirar 136 preguntas con marcadores Group A/B, el conjunto efectivo contiene 32.864 preguntas, no 33.000. Diez modelos contestan cada pregunta tres veces en conversaciones independientes, produciendo 985.920 respuestas crudas. Para cada combinación modelo–pregunta, BiasLens aplica mayoría de dos de tres: Yes y Choice usan reglas predefinidas; Why usa tres votos independientes de GPT-4o-mini por respuesta y después mayoría entre las tres respuestas. Los 107.580 casos informados no son respuestas individuales, sino combinaciones modelo–pregunta clasificadas como sesgadas entre 328.640 posibles: 32,73% dentro de este test adversarial. Los recuentos por modelo oscilan entre 7.579 para GLM-4.5 y 16.963 para Llama-3-8B. Al eliminar el prefijo de rol de las mismas preguntas, el total publicado baja de 107.580 a 81.497: 24,80% de los casos y 7,94 puntos porcentuales menos; el artículo informa una reducción relativa media por modelo de 23,8%. Este contraste pareado es el resultado más útil: en preguntas diseñadas expresamente para provocar estereotipos, añadir estos roles aumenta de forma consistente las detecciones en los diez sistemas. No es una estimación de prevalencia en uso real. Los roles se generan pidiendo grupos con mayor probabilidad de discriminar y las preguntas se redactan para elicitar sesgo; categorías como character, social y victim tampoco son atributos demográficos homogéneos. La validez del oráculo es desigual. En muestras humanas, la regla universal No para Yes/No coincide en 94,6% y la última opción para Choice en 94,4%; los ítems cuyo oráculo es incorrecto no se eliminan del total. Para Why, el juez coincide con humanos solo en 80,7% de 384 pares, y estas preguntas aportan 54.538 detecciones. La publicación destaca 2,6% de sesgos omitidos, pero no libera la matriz de confusión necesaria para explicar el 19,3% de desacuerdo. La auditoría del repositorio encuentra un error adicional: el parser Choice considera neutral cualquier texto que contenga una C mayúscula. Respuestas como A) Children y A. College graduates se clasifican como opción C. Una relectura conservadora por opción inicial cambia 6.383 casos con rol; en Qwen1.5-110B, el recuento pasa de 7.754 a aproximadamente 13.278. Esta recalculación no sustituye una nueva anotación humana, pero invalida comparaciones exactas y rankings basados en el parser actual. El efecto agregado de quitar el rol permanece descriptivamente grande porque el mismo defecto afecta ambas condiciones. El artículo usa además un z-test de dos proporciones como si las muestras fueran independientes, aunque cada pregunta se observa con y sin rol; corresponde un análisis pareado. El artefacto publica casi dos millones de respuestas, preguntas y resultados, lo que aporta mucho valor, pero no es autoconsistente: las respuestas GLM actuales regeneran 22 detecciones menos con rol y 19 menos sin rol que las tablas; faltan anotaciones humanas, generador de roles y el paso que construye final; varios scripts apuntan a carpetas inexistentes, no hay versiones fijadas, tests, CI ni archivo LICENSE. La conclusión defendible es que ciertos prefijos de rol amplifican detecciones de sesgo en un banco adversarial, no que el 32,73% de las interacciones reales sea sesgado ni que el ranking publicado mida de forma estable la equidad general de los modelos.

Research question

Can an automated framework generate roles, questions, and oracles that expose socially biased responses from LLMs during role-playing, and how much do detections change when role assignment is removed?

Method

BiasLens uses GPT-4o to generate 550 roles and adversarial Yes/No, Choice, and Why questions in 11 categories. Ten LLMs answer 32,864 questions three times with default configuration, both with role and without the role prefix. The rules consider Yes biased in Yes/No and any option other than the last in Choice; Why responses are judged with repeated votes from GPT-4o-mini. Two expert annotators and one arbiter evaluate samples of roles, questions, oracles, and categories. The article compares proportions with a z-test and studies consistency across three rounds.

Sample: 32,864 retained questions: 10,975 Yes/No, 10,917 Choice, and 10,972 Why. Ten models and three rounds produce 985,920 raw responses per condition, or 328,640 model-question combinations after majority. Human evaluation: 372 questions of each closed format, 384 Why pairs, 380 questions for bias types, and 372 questions per format for naturalness; the 550 roles are manually reviewed.

Findings

  • The published total of 107,580 equals 32.73% of 328,640 model-question combinations with role, not 107,580 independent raw responses.
  • Llama-3-8B registers the highest published count, 16,963 cases, and GLM-4.5 the lowest, 7,579; these comparisons are affected by parser errors and artifact drift.
  • Without the role prefix, the total drops to 81,497, 24.80%; the aggregate difference is 7.94 percentage points and all models decrease descriptively.
  • The article reports a mean relative reduction per model of 23.8% when removing the role.
  • 60.7% of the questions that activate some detection do so in more than three models and 14.0% in all ten, according to the published scores.
  • Culture and race concentrate the highest mean published counts among the 11 categories; the analysis is adversarial exposure, not population prevalence.
  • Responses maintain the same bias/no-bias label across the three rounds for 97.1% of Yes/No, 83.2% of Choice, and 79.3% of Why on average.
  • Human samples rate 92.8% of the questions as good in relevance and 96.4% as good in clarity by both annotators.
  • The Yes/No rule matches humans in 94.6%, Choice in 94.4%, and the Why judge in 80.7% of the sampled pairs.
  • The repository contains all expected volumes of CSV: 330 files and 985,920 responses per condition, with no duplicate keys in the audit.

Limitations

  • The bank is deliberately adversarial: roles and questions are generated to maximize the exposure of stereotypes, so the rates do not estimate frequency in natural conversations.
  • The 11 categories mix demographic attributes, occupations, character, social status, and victimization; they do not form a uniform framework of protected groups.
  • The closed rules are incorrect in approximately 5.4% of Yes/No and 5.6% of Choice according to the human sample itself, but the items are not corrected before the main total.
  • The Why judge has 80.7% global agreement with humans; no confusion matrix, labels, exact selection, or adjudications are published.
  • The Choice parser searches for an uppercase C in any position, confusing content such as Children or College with the neutral option C and altering thousands of cases.
  • The Yes/No parser searches for substrings yes and no; errors, empty responses, and out-of-format explanations can become labels without an explicit failure category.
  • The with/without role comparison is paired by question and model, but uses a z-test for two independent proportions; it does not model pairing, clustering by question/role, or multiplicity across ten models.
  • The claim of absence of a capability-bias correlation presents no coefficient, uncertainty, or test; it relies on mutable rankings and order examples.
  • The ten systems are served through OpenAI, DeepSeek, OpenRouter, and AIMLAPI with default configurations and mutable aliases; model, host, template, and date are confounded.
  • Temperature, seed, weight digest, API version, and date are not fixed for each response file; three rounds characterize variation but do not guarantee repeatability.
  • Current raw GLM data do not regenerate their saved tables and 29 Llama-3-70B records do not match exactly across rounds.
  • The role generator, the step that creates the final column, human annotations, and validation matrices are missing; several scripts have partial paths and configurations that are not executable.
  • There is no lock, dependency versions, tests, CI, checksums, release, or LICENSE file, although the README declares MIT.
  • Bias type validation counts as correct any automatic prediction that contains all human labels, without penalizing additional labels; the 93.68% is not exact set match.

What the study does not establish

  • It does not establish that 32.73% of ordinary interactions with LLMs are biased; the figure belongs to adversarial prompts selected to provoke bias.
  • It does not measure global fairness of a model nor allow a stable ranking among providers while the Choice parser and service versions are not corrected and fixed.
  • It does not demonstrate that the 107,580 cases are independent responses; they are majority decisions per model-question combination built from three responses.
  • It does not perfectly validate the oracles: human disagreement is 5-6% for closed formats and 19.3% for Why.
  • It does not prove general causality of role-playing; it identifies the effect of concrete prefixes on the same adversarial questions and under concrete providers.
  • It does not demonstrate that capability and fairness lack a relationship or that they can be optimized simultaneously; it does not perform a correlation or trade-off analysis.
  • It does not demonstrate generalization to other roles, languages, tasks, multi-turn conversations, or real deployments.
  • It does not allow end-to-end reconstruction of all tables from a clean clone with the current scripts and data.

Traceability

Scope: Full text

Version: arXiv:2411.00585v2; published in Proceedings of the ACM on Software Engineering 3 (FSE 2026), 2211-2234, DOI 10.1145/3808106

Consulted source: https://arxiv.org/pdf/2411.00585

Review: Codex 24-page full-text visual, TeX, publication, data-quality, oracle, parser, statistical, repository, executable-artifact, reproducibility and claim audit, 2026-07-16

Approval: Codex fidelity pass, 2026-07-16

English translation: approved, 2026-07-18

Models evaluated

  • GPT-4o-mini
  • GPT-5-mini
  • Qwen1.5-110B-Chat
  • Qwen3-235B-A22B-Instruct-2507
  • Llama-3-8B-Chat
  • Llama-3-70B-Chat
  • Gemini-2.5-Flash
  • GLM-4.5
  • DeepSeek-V2.5 / deepseek-chat endpoint
  • Mistral-7B-Instruct-v0.3
  • GPT-4o question generator
  • GPT-4o-mini test-oracle judge

Instruments and metrics

  • BiasLens role and adversarial question generator
  • Rule-based Yes/No oracle with No predefined as unbiased
  • Rule-based Choice oracle with the last option predefined as unbiased
  • Repeated GPT-4o-mini majority judge for Why responses
  • Three independent response trials per model-question pair
  • Three-point human relevance and clarity scale
  • Cohen's kappa for annotator agreement
  • Role-present versus role-removed prompt comparison
  • Two-proportion z-test
  • DataMuse pairwise lexical-semantic similarity

Data used

  • BiasLens benchmark: 550 roles and 32,864 retained adversarial questions
  • 985,920 raw role-conditioned model responses
  • 985,920 raw role-removed model responses
  • Saved per-model majority classifications and analysis tables
  • Unreleased human validation labels and adjudications

Evidence and location

  • Publication, DOI, pages, and corrected official authorship: Crossref DOI 10.1145/3808106 and FSE 2026 official program, checked 2026-07-16
  • Generation of roles/questions, oracles, and three-round design: arXiv 2411.00585v2, Sections 3 and 4
  • Effective size, 985,920 responses, 107,580 majorities, and per-model results: arXiv v2, Sections 4.3 and 5.1, Tables 2-3
  • Human validation of roles, questions, oracles, and bias types: arXiv v2, Sections 5.2-5.4 and Table 4
  • Effect of removing the role and consistency across rounds: arXiv v2, Sections 5.5-5.6 and Tables 5-6
  • CSV integrity, Choice parser, GLM drift, broken paths, and missing artifacts: Official BiasLens repository at commit 7c655ee4f8e7462547e10962105aa9577e8173d2
  • Consolidated audit of data, oracles, statistics, code, and reproducibility: reports/verification/article-288-fse-biaslens-adversarial-prevalence-oracle-parser-pairing-artifact-and-claim-audit.json