This FSE 2026 paper presents BiasLens, an adversarial test generator for exposing socially biased responses when an LLM is assigned a role. GPT-4o generates 50 roles for each of 11 categories and 20 Yes/No, 20 Choice, and 20 Why questions per role. After removing 136 questions containing Group A/B placeholders, the effective set contains 32,864 questions rather than 33,000. Ten models answer every question three times in independent conversations, producing 985,920 raw responses. For each model-question combination, BiasLens uses a two-of-three majority: Yes and Choice use predefined rules, while Why uses three independent GPT-4o-mini votes per response followed by a majority across the three responses. The reported 107,580 cases are not individual responses. They are model-question combinations classified as biased among 328,640 possible combinations, or 32.73% within this adversarial test. Per-model counts range from 7,579 for GLM-4.5 to 16,963 for Llama-3-8B. Removing the role prefix from the same questions lowers the published total from 107,580 to 81,497, or 24.80% of cases and an absolute reduction of 7.94 percentage points. The paper reports a mean model-level relative reduction of 23.8%. This paired contrast is the most useful result: for questions expressly designed to trigger stereotypes, adding these roles consistently increases detections across all ten systems. It is not an estimate of prevalence in ordinary use. Roles are generated by requesting groups likely to discriminate and questions are written to elicit bias; categories such as character, social, and victim are not homogeneous demographic attributes either. Oracle validity is uneven. In human samples, the universal No rule for Yes/No agrees in 94.6% of cases and the final-option Choice rule in 94.4%; items with an invalid oracle are not removed from the headline total. For Why, the judge agrees with humans on only 80.7% of 384 sampled pairs, and Why questions contribute 54,538 detections. The paper highlights a 2.6% missed-bias rate but does not release the confusion matrix needed to explain the total 19.3% disagreement. The repository audit finds an additional defect: the Choice parser treats any text containing an uppercase C as neutral. Answers such as A) Children and A. College graduates are therefore read as option C. A conservative first-option reparse changes 6,383 role-conditioned model-question cases. For Qwen1.5-110B, the count changes from 7,754 to approximately 13,278. This audit reparse is not a substitute for new human annotation, but it invalidates exact parser-based comparisons and rankings. The aggregate role-removal effect remains descriptively large because the same defect affects both conditions. The paper also applies a two-proportion z-test as if the samples were independent even though each question is observed with and without the role; a paired analysis is required. The artifact releases nearly two million responses, questions, and result files, which is valuable, but it is not self-consistent. Current GLM raw responses regenerate 22 fewer detections with roles and 19 fewer without roles than the saved tables. Human annotations, the role generator, and the code that creates the final judge label are absent; several scripts target nonexistent directories, and there are no pinned versions, tests, CI, or LICENSE file. The defensible conclusion is that specific role prefixes amplify bias detections in an adversarial test bank, not that 32.73% of real interactions are biased or that the published ranking stably measures general model fairness.
Research question
Can an automated framework generate roles, questions, and oracles that expose socially biased responses from LLMs during role-playing, and how much do detections change when role assignment is removed?