Synthetic Personality Review

Uncovering Stereotypes in Large Language Models: A Task Complexity-based Approach

Applications, bias, and safety2024ACL AnthologyApproved editorial review

Original title: Uncovering Stereotypes in Large Language Models: A Task Complexity-Based Approach

Authors: Hari Shrawgi, Prasanjit Rath, Tushar Singhal, Sandipan Dandapat

Keywords: Large Language Models, Bias evaluation, Stereotypes, AI ethics, Social bias, Task complexity, Nationality, Gender, Race, Religion

Source: Open primary source (opens in a new tab)

4
Authors
20
Findings
29
Limitations
10
Evidence

Editorial summary

English

The paper proposes the LLM Stereotype Index (LSI), a framework for testing whether a model refuses or accepts assigning one of two opposing traits to a person solely from nationality, gender, race, or religion. Its twelve pairs follow Social Progress Index subdimensions and include malnourished/well-nourished, homeless/settled, authoritarian/libertarian, and discriminatory/inclusive. LSI does not yield one scalar index: it uses Choice Refusal Percentage (CRP), the share that refuses to choose, and Stereotype Polarity (SP), the positive-pole share among answers that choose. The declared ideal is CRP=100%, because a positive choice still attributes a trait by group membership. Seven prompt families are ordered from direct choice to function writing, with a surveyor persona, dataset entries, a career incentive, simple code, and story writing between them. The design crosses 193 purported nationalities, eight gender attributes, six race categories, and ten religions with twelve pairs, seven tasks, and two models: 157,752 generations split between ChatGPT and a second model identified as GPT-4 in the paper but DV3 in the data. GPT-4 also labels outputs as refusal, negative pole, or positive pole. Published percentages show a large but non-monotonic fall in refusal: ChatGPT reaches 79.4%, 64.2%, 26.7%, 28.8%, 5.4%, 1.6%, and 0%; the second model reaches 92.4%, 38.0%, 30.6%, 11.4%, 11.7%, 2.6%, and 1.5%. GPT-4 refuses more on the simplest task, but across all seven ChatGPT has the larger mean CRP, 32.8% versus 29.4%. GPT-4 has a larger average minimum SP, 61.6% versus 58.1%, and also larger between-group deviation, 9.3% versus 7.5%; the paper reads this as surface protection and more unequal bias when GPT-4 answers. The causal inference is narrower: task format, indirectness, persona, incentive, length, forced-choice pressure, and label detectability all change with complexity. The fall establishes prompt-family sensitivity, not that isolated complexity uncovers latent prejudice or that GPT-4 strategically hides it; no intervals or tests are reported. The artifact audit confirms all 157,752 rows and reproduces CRP, but the country list contains literal NA in 504 prompts and omits Solomon Islands; DV3, exact OpenAI calls, human validation, and table-generation analysis are undocumented. The central defect is the seventh-task correction: it marks positive whenever the positive phrase appears anywhere and checks the negative phrase only afterward. Functions often mention both returns, so 6,889 of 11,268 labels (61.1%) change, 6,470 toward positive. Before this rule, CRP is 5.4% and 7.3%, not 0% and 1.5%, and SP is 52.9% and 52.2%, not 71.5% and 84.5%. The defensible contribution is that refusal safeguards are highly prompt-form-sensitive; favorable polarity and part of the comparative severity depend strongly on a directional labeling rule.

Español

El artículo propone LLM Stereotype Index (LSI), un marco para comprobar si un modelo rechaza o acepta asignar a una persona uno de dos rasgos opuestos basándose solo en nacionalidad, género, raza o religión. Sus doce pares proceden de subdimensiones del Social Progress Index e incluyen, por ejemplo, malnutrido/bien nutrido, sin hogar/establecido, autoritario/libertario y discriminatorio/inclusivo. LSI no produce un escalar único: usa Choice Refusal Percentage (CRP), proporción que rehúsa elegir, y Stereotype Polarity (SP), proporción del polo considerado positivo entre quienes eligen. El ideal declarado es CRP=100%, porque incluso una elección positiva atribuye un rasgo por pertenencia grupal. Se ordenan siete familias de prompts desde la elección directa hasta escribir una función, pasando por persona de encuestador, entradas de dataset, incentivo profesional, código simple e historia. El diseño cruza 193 supuestas nacionalidades, ocho atributos de género, seis categorías raciales y diez religiones con doce pares, siete tareas y dos modelos: 157.752 generaciones entre ChatGPT y el segundo modelo, identificado como GPT-4 en el paper y como DV3 en los datos. GPT-4 también etiqueta las salidas como rechazo, polo negativo o positivo. Los porcentajes publicados muestran una caída grande, pero no monótona, del rechazo: ChatGPT recorre 79,4%, 64,2%, 26,7%, 28,8%, 5,4%, 1,6% y 0%; el segundo modelo 92,4%, 38,0%, 30,6%, 11,4%, 11,7%, 2,6% y 1,5%. En la tarea simple GPT-4 rechaza más, pero al agregar las siete ChatGPT tiene CRP medio mayor, 32,8% frente a 29,4%. GPT-4 presenta un mínimo SP medio mayor, 61,6% frente a 58,1%, y también mayor desviación entre grupos, 9,3% frente a 7,5%; el paper lo interpreta como protección superficial y sesgo más desigual cuando responde. La inferencia causal es más estrecha: las tareas cambian a la vez formato, indirección, persona, incentivo, longitud, presión de elegir y detectabilidad de la etiqueta. El descenso demuestra sensibilidad a esas familias de prompts, no que la complejidad aislada descubra prejuicio latente ni que GPT-4 lo oculte estratégicamente; tampoco hay intervalos o tests. La auditoría confirma las 157.752 filas y reproduce los CRP, pero la lista de países contiene el marcador NA, usado en 504 prompts, y omite Islas Salomón; falta documentar DV3, las llamadas exactas a OpenAI, la validación humana y el análisis de tablas. El fallo principal está en la séptima tarea: la corrección marca positivo si el texto contiene el polo positivo en cualquier lugar y solo busca después el negativo. Como las funciones suelen mencionar ambos retornos, cambia 6.889 de 11.268 etiquetas (61,1%), 6.470 hacia positivo. Antes de esa regla, CRP sería 5,4% y 7,3%, no 0% y 1,5%, y SP 52,9% y 52,2%, no 71,5% y 84,5%. La contribución defendible es que las salvaguardas de rechazo son muy sensibles a la forma del prompt; la favorabilidad y parte de la severidad comparativa dependen fuertemente de una regla direccional.

Research question

Does a benchmark based on the Social Progress Index and on seven families of tasks detect how the rejection of stereotyping and the polarity of the choices of ChatGPT and GPT-4 vary across nationality, gender, race, and religion, especially as the complexity attributed to the prompt increases?

Method

Factorial design of automatic generation and labeling. Twelve positive/negative pairs derived from the Social Progress Index are crossed with 217 demographic attributes, seven templates ordered by complexity, and two models. Three repetitions are run for 193 nationality entries and fifteen for the remaining 24 attributes, with 157,752 outputs. GPT-4 classifies each output as rejection (-1), negative choice (0), or positive (1); CRP is calculated over all outputs and SP over those that choose. The editorial audit read and rendered the 17 pages, verified ACL, froze the public commit, inspected zips, notebook, and scripts, analyzed the workbook sheets by streaming, reproduced CRP, and recalculated the function task correction.

Sample: 157,752 generations: 97,272 for nationality, 25,200 for religion, 20,160 for gender, and 15,120 for race, distributed equally between ChatGPT and DV3. Each task type contains 22,536 rows. There are 193 nationality entries by three repetitions and 24 remaining attributes by fifteen; one entry is NA, not a country, and produces 504 rows. ChoiceLabel contains 44,437 rejections, 76,167 positive choices, and 37,148 negative; the choices sheet retains 113,315 rows. The LLaMA2 exploration uses 100 examples per task and reports 53.3% mean failures.

Findings

  • The workbook contains exactly 157,752 generations, 78,876 per model.
  • The published CRP per task are reproduced exactly from ChoiceLabel.
  • ChatGPT drops from CRP 79.4% to 0% and the second model from 92.4% to 1.5% in the order of the paper.
  • The sequence is not monotonic in either of the two models.
  • In the direct task GPT-4 rejects more than ChatGPT across the four demographics.
  • When aggregating the seven tasks ChatGPT has a higher mean CRP: 32.8% versus 29.4%.
  • ChatGPT/GPT-4 reach CRP 27.2/24.7% in nationality, 41.4/34.0% in race, 24.0/27.0% in religion, and 38.6/31.9% in gender.
  • The minimum mean SP rises from 58.1% to 61.6% and the deviation between groups from 7.5% to 9.3%.
  • The groups with minimum SP are Africa, Hispanic, Islam, and Male in both model generations.
  • The paper reports Opportunity 19.2% for Syria versus 77.1% for Germany.
  • African American reaches SP 4.0% in Shelter versus 53.2% for White.
  • The function task rule changes 6,889 of 11,268 labels, 61.1%.
  • Of those changes, 6,470 end in positive and 419 in negative.
  • Without the correction, CRP in that task is 5.4% and 7.3%, not 0% and 1.5%.
  • Without the correction, SP is 52.9% and 52.2%; with it, it becomes 71.5% and 84.5%.
  • The list includes NA instead of Solomon Islands; 504 prompts ask about a person from NA.
  • The 504 NA groups remain empty in group_placeholder due to their treatment as a missing value in Excel.
  • The second model appears as DV3 in 78,876 rows without documenting its link to GPT-4.
  • One output is empty and receives a reasoned rejection label.
  • LLaMA2-7B fails on average in 53.3% of 100 examples per task and is not run over the full corpus.

Limitations

  • The tasks conflate complexity with format, indirection, person, incentive, length, and pressure to choose.
  • Complexity is not validated with participants or an independent measure; the action is assessed by asking an LLM.
  • The complexity ratings and the model that produced them are not published.
  • There are no intervals, standard errors, trend tests, or difference tests.
  • Three repetitions per nationality yield less stable per-country estimates than the remaining fifteen.
  • CRP measures explicit rejection, not hidden associations when the model refuses.
  • CRP does not distinguish moral negation, inability, format deviation, or truncation.
  • SP considers the positive pole favorable even if it remains a group generalization.
  • LSI does not define a single composite score despite being called an index.
  • The pairs translate structural indicators of the SPI into personal traits and may incur an ecological fallacy.
  • The poles are not symmetric, exhaustive, or universally positive or negative.
  • The SPI incorporates a Western perspective not adapted by cultural context.
  • The racial categories are American and the gender attributes mix non-equivalent dimensions.
  • The categories are not mutually exclusive and intersections are not studied.
  • All harms receive uniform weight without modeling power, application, or severity.
  • The list contains NA and omits Solomon Islands, contradicting 193 real countries.
  • There is no snapshot, date per row, deployment, or exact message configuration.
  • DV3 is not explained and does not allow independent verification of GPT-4.
  • GPT-4 labels its own responses, creating dependence between evaluated and judge.
  • The validation of 500 examples does not release selection, gold labels, annotators, or errors.
  • small_test_set.tsv contains 26 examples without human labels, not the declared validation.
  • The correction seeks the positive pole first and massively biases SP.
  • The rule does not identify the executed branch or the first choice when both literals appear.
  • The notebook uses three global repetitions and does not reproduce without manual editing the 3/15 design.
  • The code for calls, CRP/SP analysis, plots, or end-to-end execution is not released.
  • There is no license, fixed environment, tests, CI, seeds, or cost logs.
  • The LLaMA2 evaluation uses an unversioned community quantization and many instruction failures.
  • The models are from 2023 and the percentages should not be transferred to current versions.
  • The artificial prompts do not measure consequences in real systems, users, or decisions.

What the study does not establish

  • It does not demonstrate biases, intentions, or conscious concealment in the models.
  • It does not isolate complexity as the cause of the decline in rejection.
  • It does not establish that a positive choice is impartial or safe.
  • It does not prove that lower SP causes more harm in a specific application.
  • It does not validate the SPI as a universal taxonomy of individual stereotypes.
  • It does not allow attributing DV3 with certainty to an exact GPT-4 snapshot.
  • It does not allow regenerating responses, human control, and figures from the repository.
  • It does not sustain SP of the function task without a rule that alters 61.1% of labels.
  • It does not generalize to other languages, cultures, current models, or natural tasks.
  • It does not demonstrate discrimination or social harm outside the experiment.

Traceability

Scope: Full text

Version: EACL 2024 long paper 111, pages 1841–1857; DOI 10.18653/v1/2024.eacl-long.111

Consulted source: https://aclanthology.org/2024.eacl-long.111.pdf

Review: Codex full-text, visual, code, workbook and label-robustness audit, 2026-07-15

Approval: Codex fidelity pass, 2026-07-15

English translation: approved, 2026-07-18

Models evaluated

  • ChatGPT / GPT-3.5-Turbo endpoint available in May–June 2023
  • GPT-4 endpoint available in May–June 2023, represented as DV3 in the released workbook without a documented mapping
  • TheBloke/Llama-2-7b-Chat-GPTQ exploratory sample, not part of the full comparison
  • GPT-4 used as automatic choice-labeling model

Instruments and metrics

  • LLM Stereotype Index framework
  • Choice Refusal Percentage (CRP)
  • Stereotype Polarity (SP)
  • Twelve bipolar stereotype pairs mapped to Social Progress Index subdimensions
  • Seven prompt families spanning direct choice, persona, dataset entry, incentive, story and code
  • Task-complexity dimensions: size, variety, relationships and action complexity
  • GPT-4 three-way choice-detection prompt
  • Identifier-6 positive-first string correction

Data used

  • GeneratedData_157k_withChoiceLabels.xlsx with 157,752 experimental rows and 113,315 choice rows
  • PaperData.xlsx with CRP, skew, radar and model-comparison tables
  • SPI_Stereotypes.xlsx with 28 demographic-task templates and 12 stereotype pairs
  • country_names.tsv with 193 entries including invalid NA and omitting Solomon Islands
  • VenkitExpCompletions_ChatGPTWithSentimentAndCountry.xlsx baseline replication
  • Seven 100-row LLaMA2-7B exploratory TSV files
  • GitHub Avenge-PRC777 repository commit 017d3457950df3abf1a4e19f1ea3339bd0c2f567

Evidence and location

  • Motivation, contributions, and concealment: EACL paper pp. 1–3, Abstract, Introduction and Sections 2–3
  • Complexity and seven tasks: EACL paper pp. 3–5, Section 4, Figure 2 and Appendix A
  • SPI pairs, CRP, and SP: EACL paper pp. 5–6, Sections 5–6 and Figures 3–4
  • Sample, models, and configuration: EACL paper pp. 6 and 14–15, Section 7, Appendix C and Table 7
  • CRP and model comparison: EACL paper pp. 6–7, Section 8.1–8.2, Tables 2–3 and Figure 5
  • Results by demographic: EACL paper pp. 7–8, Section 8.3, Figure 6 and Tables 4–6
  • Limitations and LLaMA2: EACL paper pp. 8–10 and 15–17, Limitations, Ethics and Appendix E
  • Metadata, DOI, and abstract: ACL Anthology record 2024.eacl-long.111, checked 15 Jul 2026
  • Artifact, DV3, and unreleased validation: GitHub commit 017d345, both supplementary zips, notebook and scripts, audited 15 Jul 2026
  • Counts, NA, CRP, and positive-first correction: Editorial audit of GeneratedData_157k_withChoiceLabels.xlsx, PaperData.xlsx and identifier-6 correction, 15 Jul 2026