Large Language Models Portray Socially Subordinate Groups as More Homogeneous, Consistent with a Bias Observed in Humans

Applications, bias, and safety2024ACMApproved editorial review

Authors: Messi H.J. Lee, Jacob M. Montgomery, Calvin K. Lai

Keywords: Large Language Models, Social bias, Homogeneity perception, Racial minorities, Fairness, Accountability

Source: Open primary source (opens in a new tab)

3
Authors
25
Findings
42
Limitations
11
Evidence

Editorial summary

English

The paper introduces representational homogeneity as a bias dimension distinct from associating groups with stereotypical attributes: it asks whether ChatGPT produces a narrower semantic repertoire for socially subordinate U.S. groups. The main study was registered on OSF before data collection and crosses four racial or ethnic labels, African, Asian, Hispanic, and White American, two genders, man and woman, and 13 short text formats. On 25 July 2023, the authors requested 500 approximately 30-word completions for each of the 104 combinations, 52,000 texts in total, using the `gpt-3.5-turbo` alias, a `chatbot` system role, and default parameters except for batches of 128, 128, 128, and 116 responses. Fifty refusals detected with six expressions were replaced; the final released dataset has exactly 500 texts per cell and no blanks. Mean length is 26.61 words (SD 2.70, range 9–68), and only 3,645 texts contain exactly 30 whitespace-separated words. The primary measure encodes each text with the penultimate layer of BERT-base-uncased and calculates all 124,750 pairwise cosine similarities among the 500 texts in each cell. It then standardizes 12,974,000 values globally and fits linear mixed models with race, gender, their interaction, and a random intercept for text format. Unregistered robustness analyses add the third-to-last BERT layer, two RoBERTa layers, and three Sentence-BERT encoders. Under BERT−2, standardized race coefficients relative to White Americans are 0.33 for African Americans, 0.31 for Asian Americans, and 0.18 for Hispanic Americans; the aggregate women-versus-men coefficient is 0.037. The BERT−2 interaction estimates women-minus-men differences of 0.0099, 0.013, and 0.12 in the three minority groups, while 0.00021 for White Americans is not different from zero under the published analysis. The subordinate-versus-White direction holds across all seven encoders and after removing race and gender words. The gender conclusion is less stable: African and Hispanic Americans retain a positive difference under all seven representations, whereas signs change by encoder for Asian and White Americans. An exploratory structural topic model finds that adversity topics dominate 41.86% of African, 26.15% of Asian, 18.65% of Hispanic, and 3.57% of White texts. Two unregistered follow-ups, 10,400 texts explicitly instructed not to mention discrimination, hardship, or adversity, and 800 stories about a chef preparing a meal for a loved one, plus within-topic comparisons for two adversity topics preserve the racial direction. This supports the claim that observed topical alignment is not the signal's only source, but it does not identify its cause. The central reservation affects all inference: the 12,974,000 rows are not independent replications. Every one of the 52,000 texts occurs in 499 pairs, so pairs sharing a text are correlated; the model includes only a format intercept and does not represent texts, dyads, or generation batches. Standard errors below 0.001, degrees of freedom near 12.97 million, z statistics in the hundreds, and follow-up tests are therefore anti-conservative. Table 2 also reports N=124,750 per intersectional group although the code combines 13 formats, 1,621,750 pairs per group, while Table 3 uses 12,974,000. The repository makes the text, prompts, and formulas auditable and is MIT-licensed, but it is not end-to-end executable as published: scripts load absent `.RData` and Feather files, no locked environments or encoder snapshots are supplied, there are no tests or CI, several paths do not match the tree, and the latest commit deleted the instructions still linked by the README. The defensible contribution is a descriptive pattern of reduced semantic diversity for three racial-minority labels in one dated ChatGPT collection, robust across several encoders and prompts. It does not establish a general LLM property, a training-data cause, equivalence to human perception, or the reported inferential precision.

Español

El artículo introduce la homogeneidad representacional como una dimensión de sesgo distinta de asociar grupos con atributos estereotípicos: pregunta si ChatGPT produce un repertorio semántico menos variado para grupos estadounidenses socialmente subordinados. El estudio principal, prerregistrado en OSF antes de recoger datos, cruza cuatro etiquetas raciales o étnicas, African, Asian, Hispanic y White American, dos géneros, man y woman, y 13 formatos breves. El 25 de julio de 2023 se solicitaron 500 completaciones de aproximadamente 30 palabras para cada una de las 104 combinaciones, 52.000 textos en total, mediante el alias `gpt-3.5-turbo`, rol de sistema `chatbot` y parámetros predeterminados salvo cuatro lotes de 128, 128, 128 y 116 respuestas. Cincuenta negativas detectadas mediante seis expresiones se sustituyeron; el dataset final conserva exactamente 500 textos por celda, sin vacíos. Su longitud media es 26,61 palabras (DE 2,70; intervalo 9–68) y solo 3.645 textos tienen exactamente 30 palabras. La medida principal codifica cada texto con la penúltima capa de BERT-base-uncased y calcula las 124.750 similitudes coseno posibles entre los 500 textos de cada celda. Después estandariza globalmente 12.974.000 valores y ajusta modelos lineales mixtos con raza, género, su interacción y un intercepto aleatorio para formato. Se añaden, sin prerregistro, BERT en la antepenúltima capa, dos capas de RoBERTa y tres Sentence-BERT. En BERT−2, frente a White Americans, los coeficientes raciales estandarizados son 0,33 para African Americans, 0,31 para Asian Americans y 0,18 para Hispanic Americans; el coeficiente agregado de women frente a men es 0,037. La interacción BERT−2 estima diferencias mujer-hombre de 0,0099, 0,013 y 0,12 en los tres grupos minoritarios, mientras que 0,00021 en White Americans no difiere de cero según el análisis publicado. La dirección subordinado-versus-White se mantiene en las siete representaciones y al eliminar del texto palabras de raza y género. La conclusión de género es más frágil: African e Hispanic Americans mantienen una diferencia positiva en las siete representaciones, pero el signo cambia según el encoder para Asian y White Americans. Un STM exploratorio encuentra que los temas de adversidad dominan el 41,86% de textos African, 26,15% Asian, 18,65% Hispanic y 3,57% White. Dos seguimientos no prerregistrados, 10.400 textos con una prohibición explícita de mencionar discriminación, hardship o adversity, y 800 historias de una persona cocinando para un ser querido, además de comparaciones dentro de dos temas de adversidad, conservan la dirección racial. Esto apoya que la alineación temática observada no es la única fuente de la señal, pero no identifica su causa. La principal reserva afecta a toda la inferencia: las 12.974.000 filas no son réplicas independientes. Cada uno de los 52.000 textos participa en 499 pares, y los pares que comparten un texto están correlacionados; el modelo solo incluye un intercepto por formato y no representa textos, díadas o generaciones. Por tanto, errores estándar inferiores a 0,001, grados de libertad cercanos a 12,97 millones, valores z de cientos y los tests de seguimiento son anticonservadores. Además, la Tabla 2 declara N=124.750 por grupo interseccional, aunque el código agrega 13 formatos, es decir, 1.621.750 pares por grupo, mientras la Tabla 3 usa 12.974.000. El repositorio permite auditar los textos, prompts y fórmulas y tiene licencia MIT, pero no es ejecutable de extremo a extremo tal como está publicado: faltan archivos `.RData` y Feather cargados por los scripts, entornos bloqueados, snapshots de encoders, pruebas y CI; varias rutas no coinciden con la estructura y el último commit eliminó las instrucciones todavía enlazadas en el README. La contribución defendible es un patrón descriptivo de menor diversidad semántica para tres etiquetas raciales minoritarias en una recogida concreta de ChatGPT, robusto a varios encoders y prompts. No demuestra una propiedad general de los LLM, una causa en los datos de entrenamiento, equivalencia con percepción humana ni la precisión inferencial publicada.

Research question

Does ChatGPT generate semantically more homogeneous texts about socially subordinated racial or ethnic groups and genders than about dominant groups, and is the gender effect consistent across racial or ethnic groups?

Method

Preregistered factorial design of 4 racial or ethnic labels × 2 genders × 13 formats, with 500 `gpt-3.5-turbo` generations per cell. All pairwise cosine similarities are computed within each cell using BERT−2 and six alternative representations, globally standardized, and mixed models are fitted with a random intercept for format. The editorial audit read and rendered the 20 pages, read and rendered the two DOCX files from the OSF preregistration, froze the public commit, reviewed code, tables, and datasets, profiled the 52,000 rows, and contrasted the grain of the data with the inferential unit.

Sample: The main study contains 52,000 texts: 104 prompt cells with 500 completions each. Each cell produces 500×499/2=124,750 similarities, for 12,974,000 pairs per encoder. The real generative unit is 52,000 texts obtained in four batches per prompt; the 12,974 million pairs are not independent observations. Follow-up 1 adds 10,400 texts, 100 per the same 104 cells; follow-up 2 adds 800 texts, 100 per each combination of four groups and two genders.

Findings

  • The published source is the FAccT 2024 paper, DOI 10.1145/3630106.3658975, 20 pages, licensed CC BY 4.0.
  • The OSF registration kxz6b was registered publicly on 26 July 2023 and contains the analytic plan and the 104 vignettes.
  • The dataset audit confirms 52,000 rows, four columns, 104 cells, and exactly 500 texts per cell, with no empty values.
  • The 50 rejections from the first batch are removed; three substitutions retrigger the filter and are replaced in a second round.
  • The final dataset contains three additional exact duplicates and four after normalizing capitalization and spaces, a rate below 0.01%.
  • The reproduced mean length is 26,610 words and the standard deviation 2,696; the observed range is 9–68 words.
  • Only 3,645 of 52,000 texts have exactly 30 words; 44,792 fall below and 3,563 above.
  • Each cell of 500 texts generates 124,750 pairs and each text reappears in 499 similarity observations.
  • BERT−2 estimates racial differences of 0.33, 0.31, and 0.18 standard deviations for African, Asian, and Hispanic Americans relative to White Americans.
  • The racial direction holds across the seven text representations examined.
  • The aggregate effect of women relative to men is 0.037 standard deviations in BERT−2 and is smaller than the published racial effects.
  • In BERT−2, the woman-man difference is positive for African, Asian, and Hispanic Americans and practically null for White Americans.
  • The gender direction is stable across encoders for African and Hispanic Americans, but changes for Asian and White Americans.
  • After removing words that signal race and gender, the racial differences are 0.34, 0.28, and 0.18, and the aggregate gender difference is 0.073.
  • Preprocessing robustness alters the interaction: it also obtains a positive difference for White American women relative to men.
  • Topics 1 and 10, associated with hardship and adversity, dominate 41.86%, 26.15%, 18.65%, and 3.57% of African, Asian, Hispanic, and White texts.
  • Table A2 itself contradicts the expectation that White texts have the most dispersed thematic distribution.
  • The follow-up without hardship reduces mentions of adversity or barrier among the three minority groups from 24.80% to 3.00% and retains the racial direction.
  • The cooking follow-up contains 800 balanced texts and retains the published racial direction.
  • Comparisons within topics 1 and 10 also maintain higher mean similarity for the three minority groups than for White Americans.
  • Table 2 reports N=124,750 per intersectional group, but the code uses 13 times that amount per group and Table 3 reports 12,974,000 observations.
  • The published degrees of freedom and standard errors treat overlapping pairs as independent residuals.
  • The repository publishes data, prompts, code, figures, and preregistration under an MIT license, enabling substantive static audit.
  • The frozen commit does not include the `.RData` or Feather files that several analysis phases load.
  • The README links Instructions.md, but the current commit removed it; the file remains accessible only in the Git history.

Limitations

  • Only a single collection of the `gpt-3.5-turbo` alias on a single date is studied.
  • The exact snapshot identifier returned by the API is not retained.
  • The model alias, default parameters, and service may change over time.
  • No request IDs, per-batch timestamps, errors, retries, or response ordering are published.
  • The 52,000 texts are not 52,000 independent runs: each request returns up to 128 options under the same API context.
  • The measure quantifies embedding similarity, not human perception of homogeneity.
  • It is not validated that cosine differences correspond to harms, stereotypes, or a narrower range of human experience perceived by people.
  • The choice of an encoder may introduce geometry and biases of its own unrelated to ChatGPT.
  • Although seven representations reinforce the racial direction, their exact checkpoints and revisions are not fixed.
  • The 12,974,000 similarities share nodes: each text appears in 499 pairs.
  • The model does not add random effects for text, dyad, batch, or individual prompt.
  • A single intercept for 13 formats does not correct the dependence between pairs that share a text.
  • Standard errors below 0.001 and p-values are anticonservative under that pseudo-replication.
  • The same dependence invalidates the conventional interpretation of t-tests and models in the follow-ups.
  • Table 2 presents an N per group incompatible with the total constructed by the code.
  • Global standardization means the coefficients depend on the complete mix of groups and formats.
  • Only explicit, binary labels in US English are used.
  • Non-binary people, Native American, Middle Eastern American, or other smaller groups are not studied.
  • Names, specific nationalities, alternative labels, or implicit identities are not tested.
  • The terms White American and man are artificially marked although they are usually unmarked in discourse.
  • Brief texts of approximately 30 words do not represent conversations, long documents, or deployed interaction.
  • Only 7.01% of texts exactly meet the requested length of 30 words.
  • The rejection filter uses six matches sensitive to capitalization and does not demonstrate exhaustive coverage of negations or reformulations.
  • The 50 replacements are generated afterward and may belong to a different state of the service.
  • The six alternative encoders and the follow-up studies were not preregistered.
  • No structured record of deviations among preregistration, confirmatory analysis, and exploratory analysis is presented.
  • The STM selects 15 topics from four candidates and the identification of adversity depends on the interpretation of top words.
  • Searching only for adversity and barrier is an incomplete check that the follow-ups remove the topic.
  • Comparisons within topics condition on a classification estimated by the same corpus and do not causally isolate content.
  • The follow-ups simultaneously modify the prompt, the content, and potentially the model behavior.
  • Training data is neither intervened upon nor its composition observed, so selection and stereotypical associations are hypotheses.
  • There is no comparison with equivalent human texts or a human diversity baseline.
  • There is no blind human evaluation of diversity, quality, naturalness, or harm.
  • No temporal, cross-provider, cross-language, or cross-LLM-family replications are performed.
  • The repository does not pin versions of R, Python, and packages via an executable lockfile.
  • The main scripts use `../data` while the published directory is called `Data`, which fails on case-sensitive systems.
  • The generation scripts build nested output paths that do not match the instructions or the published tree.
  • Several scripts unconditionally load absent `.RData` files after leaving the line that would save them commented out.
  • The three Sentence-BERT Feather files are not published and the notebook requires manually changing the model between runs.
  • There are no automated tests, CI, container, internal checksums, or a documented clean run.
  • The current README links an instructions document removed in the last commit.
  • The README claim that all necessary code and data are available does not amount to end-to-end reproduction in the frozen state.

What the study does not establish

  • It does not demonstrate that all LLMs represent subordinate groups as more homogeneous.
  • It does not demonstrate that the behavior persists in current versions of ChatGPT.
  • It does not establish that social status is the cause of the observed differences.
  • It does not causally identify the origin of the pattern in the training data.
  • It does not prove equivalence between embedding similarity and homogeneity perceived by humans.
  • It does not demonstrate that users adopt stereotypes or discriminate based on those generations.
  • It does not demonstrate that thematic alignment has been entirely ruled out.
  • It does not support treating 12,974 million pairs as independent replicates.
  • It does not validate the magnitude or the published inferential intervals under dyadic dependence.
  • It does not generalize to long text, other prompts, languages, cultures, identities, or deployed contexts.
  • It does not establish that explicitly mentioning dominant groups is a neutral control.
  • It does not yet provide an executable end-to-end reproduction from a clean environment.

Traceability

Scope: Full text

Version: FAccT 2024 paper 90; proceedings pages 1321–1340; DOI 10.1145/3630106.3658975; CC BY 4.0

Consulted source: https://facctconference.org/static/papers24/facct24-90.pdf

Review: Codex full-text, visual, preregistration, repository, code, data-quality and statistical-grain audit, 2026-07-15

Approval: Codex fidelity pass, 2026-07-15

English translation: approved, 2026-07-18

Models evaluated

  • OpenAI gpt-3.5-turbo alias accessed 25 July 2023; exact returned snapshot not preserved
  • BERT-base-uncased penultimate layer (preregistered primary representation; exact revision not pinned)
  • BERT-base-uncased third-to-last layer
  • RoBERTa-base penultimate layer
  • RoBERTa-base third-to-last layer
  • sentence-transformers/all-mpnet-base-v2
  • sentence-transformers/all-distilroberta-v1
  • sentence-transformers/all-MiniLM-L12-v2
  • 15-topic structural topic model

Instruments and metrics

  • 104 prompts crossing four racial or ethnic labels, two gender labels and 13 text formats
  • Thirty-word generation instruction
  • Pairwise cosine similarity of sentence embeddings as representational homogeneity
  • Globally standardized cosine similarity
  • Linear mixed-effects models with a random intercept for text format
  • Likelihood-ratio tests for race, gender and interaction terms
  • Estimated marginal means for within-race gender comparisons
  • Identity-term removal robustness analysis
  • Structural topic model with 15 topics
  • No-hardship follow-up prompt
  • Chef preparing a meal follow-up prompt

Data used

  • Released main dataset generated_text_final.csv, 52,000 texts
  • Four original generation batches totaling 52,000 texts
  • Fifty first-round and three second-round replacement generations
  • Suppression study 1 dataset, 10,400 texts
  • Suppression study 2 dataset, 800 texts
  • Suppression pilot dataset, 1,040 texts
  • Released Topic 1 and Topic 10 subsets
  • OSF registration kxz6b and two frozen preregistration documents
  • GitHub repository at commit a08b419ccec38136aa977b4ccf21b8901445d364

Evidence and location

  • Question, design, collection, and main measure: FAccT 2024 pp. 1321–1324, Abstract and Sections 1–2
  • Racial, gender, and interaction results: FAccT 2024 pp. 1324–1326, Figures 1–4 and Tables 2–3
  • Discussion, acknowledged limitations, and conclusion: FAccT 2024 pp. 1326–1327, Sections 4–6
  • Adversity, cooking follow-ups, and within-topic analyses: FAccT 2024 pp. 1329–1331, Supplement A.2
  • Thematic distribution, rejections, and preprocessing: FAccT 2024 pp. 1331–1332, Supplement A.3–A.5
  • Robustness across seven representations: FAccT 2024 pp. 1333–1340, Tables A3–A10
  • Confirmatory plan and 104 vignettes: OSF registration kxz6b, registered 26 Jul 2023; preregistration pp. 1–3 and vignettes pp. 1–2
  • Grain of 12,974,000 pairs and model with intercept only by format: Frozen repository commit a08b419, BERT-2/bert2.R lines 79–231; compared with Table 3
  • Quality profile of 52,000 texts, balance, duplicates, and lengths: Frozen generated_text_final.csv, independently profiled 15 Jul 2026
  • Replacement of negatives and follow-up datasets: Frozen repository commit a08b419, Data/merge_texts.R and Supplement.Suppression datasets and scripts
  • Execution gaps, paths, artifacts, and documentation: Frozen repository commit a08b419 tree and code; Instructions.md recovered from parent commit 4f85271; audited 15 Jul 2026