“I’ve never seen a glass ceiling better represented”: Bias and gendering in LLM-generated synthetic personas from a participatory design perspective

Applications, bias, and safety2025ElsevierApproved editorial review

Authors: Helena A. Haxvig, Vincenzo D’Andrea, Maurizio Teli

Keywords: Large Language Models, Personality, Bias, Social Impact, Evaluation

Source: Open primary source (opens in a new tab)

3
Authors
8
Findings
14
Limitations
4
Evidence

Editorial summary

English

This paper examines how ChatGPT 4o, Gemini 1.5 Flash, Cohere Command, and Pi.ai generate gendered synthetic personas, and what participatory evaluation adds to expert counting. Drawing on feminist theory, queer HCI, performativity, and Participatory Design, it does not treat bias as deviation from one neutral truth: it studies how names, labels, occupations, ambitions, traits, and social positions perform gender. Its main conclusion is that synthetic personas should not replace real users, although they may be useful provocations for exposing assumptions.

The design combines three direct inquiries with a pilot and two workshops. In November 2024, the authors requested 20 personas for each combination of four models and “Gender,” “Gender Identity,” or “Pronouns,” and another 20 for each model and eleven products. In January 2025, 50 names with binary Demografix gender probabilities between 50% and 70% were crossed with the same three fields and four models. The public workbooks contain 240, 880, and 600 rows: 1,720 personas. Each condition, however, appears to be one batched request, giving 68 calls, 12, 44, and 12, rather than 1,720 independent executions. Without repeated calls or between-run variance, the frequencies describe these batches, not stable model rates.

In inquiry 1, all four “Gender” batches were almost entirely binary. Under “Gender Identity,” ChatGPT produced six women, two men, two non-binary, six genderqueer, and four agender personas; Gemini and Cohere remained ten women and ten men, while Pi produced ten women, nine men, and one non-binary persona. Under “Pronouns,” ChatGPT included seven “they,” Gemini remained ten “he” and ten “she,” Cohere added two “she/they,” and Pi one “they.” Wording broadened labels in some batches, but neither uniformly nor as a general solution to bias.

Across ten products not biologically targeted in inquiry 2, outputs were usually close to ten women and ten men. For tampons, ChatGPT generated eighteen women and two non-binary personas; Gemini ten women, nine men, and one non-binary; Cohere ten women and ten men; and Pi twenty women. The paper also finds more female associations with creative, freelance, and care work, warmth, and empathy, and more male associations with technical or stable work, ambition, analysis, and career. These are descriptive counts, not causal mechanisms.

Inquiry 3 exposes the limits of calling names “neutral” when selected through a binary API. ChatGPT generated broader labels; Gemini remained almost entirely binary, and Cohere broadened only modestly. Pi had the same distribution under “Gender” and “Gender Identity.” Audit of the files shows that the two 50-row blocks match in 49 complete profiles and 249 of 250 cells across name, age, gender, location, and occupation; only one occupation is missing in the second block. The artifacts cannot distinguish model repetition from copying or processing, but these are not independent realizations.

The workbooks use COUNTIFS, exact occupation labels, and adjective substring searches after manual normalization. Thresholds of 15, 20, or 25 filter presentation; they are not preregistered statistical criteria. No inferential tests, effect sizes, intervals, or sensitivity analyses are reported. Products, prompts, batch size, semicolon CSV, and normalization choices structure the results. Co-occurrence does not itself identify a training-data cause.

The workshops enrolled eight self-selected volunteers: seven women and one man; four European, one South-American, one Middle Eastern, and two Asian participants. Age and LLM-experience tables each sum to seven despite N=8, without an explicit missingness note. All chose ChatGPT although four interfaces were offered. They reacted to a persona built from their name, discussed judging others, co-created personas, and conversed with them. Twenty-seven co-created personas are reported, but tables sum to 27 ages, 25 locations, and 26 genders. Audio was anonymized during transcription and deleted; full transcripts are not shared.

The qualitative analysis combines descriptive, in-vivo, concept, and affective coding with a provisional theoretical codebook, then focused and pattern coding. All three authors coded an initial subset with 92% agreement and kappa 0.48, and another with 98% and kappa 0.85; the first author coded the rest. Subset sizes, units, prevalence, tables, and calculations are omitted. Participants expressed frustration, amusement, or unease, questioned model neutrality, and identified associations aggregates could hide. The public appendix contains eighteen debate personas, eight selected and ten residual, and shows how an explicit attempt at diversity may still yield typified archetypes.

The evidence supports a bounded conclusion: in these late-2024 and early-2025 batches, outputs were predominantly binary and associated gender with work, ambition, and personality. Explicit identity or pronoun fields broadened labels in some cases, especially ChatGPT, but did not remove stereotypes. The study does not establish which current model is less biased, that 1,720 rows are independent trials, that inclusive prompting solves bias, that co-occurrences have an identified cause, or that eight participants represent a broad population.

Descriptive reproducibility is better than usual: the open final PDF, three XLSX files with prompts, rows, formulas, and tables, and the eleven-page appendix were recovered. There are no hidden sheets, defined names, or spreadsheet error cells. Exact snapshots, parameters, seeds, raw chats, response IDs, per-call timestamps, an executable environment, and explicit row provenance are missing. Two appendices named in the paper, transcripts, the coding workbook, and reliability calculations were not recovered. The responsible use is to interrogate bias with these personas, never to substitute them for research with real people.

Español

Este artículo estudia cómo ChatGPT 4o, Gemini 1.5 Flash, Cohere Command y Pi.ai generan personas sintéticas marcadas por el género, y qué añade una evaluación participativa a los recuentos de expertos. Desde teoría feminista, HCI queer, performatividad y Diseño Participativo, no trata el sesgo como una desviación frente a una única verdad neutral: observa cómo nombres, etiquetas, ocupaciones, ambiciones, rasgos y posiciones sociales producen género. Su conclusión principal es que estas personas no deben sustituir a usuarios reales, aunque pueden servir como provocaciones para hacer visibles supuestos.

El diseño combina tres consultas directas con un piloto y dos talleres. En noviembre de 2024 se solicitaron 20 personas por combinación de cuatro modelos y los campos «Gender», «Gender Identity» o «Pronouns», y otras 20 por combinación de modelo y once productos. En enero de 2025 se usaron 50 nombres cuya probabilidad binaria de género en Demografix estaba entre 50 % y 70 %, cruzados con los mismos tres campos y cuatro modelos. Los libros públicos contienen 240, 880 y 600 filas: 1.720 personas. Pero cada condición parece una sola petición por lotes, de modo que hay 68 llamadas, 12, 44 y 12, y no 1.720 ejecuciones independientes. Sin repeticiones ni varianza entre llamadas, las frecuencias describen estos lotes, no tasas estables de cada modelo.

En la primera consulta, los cuatro lotes con «Gender» fueron casi totalmente binarios. Con «Gender Identity», ChatGPT produjo seis mujeres, dos hombres, dos personas no binarias, seis genderqueer y cuatro agender; Gemini y Cohere siguieron con diez mujeres y diez hombres, y Pi con diez mujeres, nueve hombres y una persona no binaria. Con «Pronouns», ChatGPT incluyó siete «they», Gemini mantuvo diez «he» y diez «she», Cohere añadió dos «she/they» y Pi un «they». El wording amplió etiquetas en algunos lotes, pero no de forma uniforme ni como solución general al sesgo.

En los diez productos no biológicamente dirigidos de la segunda consulta predominó un reparto cercano a diez mujeres y diez hombres. Para tampones, ChatGPT generó 18 mujeres y dos personas no binarias; Gemini diez mujeres, nueve hombres y una no binaria; Cohere diez mujeres y diez hombres; y Pi veinte mujeres. El artículo encuentra además más asociaciones femeninas con ocupaciones creativas, freelance y de cuidados, calidez y empatía; las masculinas con funciones técnicas o estables, ambición, análisis y carrera. Estos son recuentos descriptivos, no mecanismos causales.

La tercera consulta muestra los límites de llamar «neutrales» a nombres seleccionados mediante una API binaria. ChatGPT generó mayor diversidad de etiquetas; Gemini siguió casi completamente binario y Cohere solo amplió de forma limitada. Pi dio el mismo reparto para «Gender» y «Gender Identity». La auditoría confirma que ambas tandas de 50 coinciden en 49 perfiles completos y en 249 de 250 celdas de nombre, edad, género, ubicación y ocupación; solo falta una ocupación en la segunda. No puede determinarse si fue repetición del modelo o un problema de copia o procesamiento, pero no son realizaciones independientes.

Las hojas usan COUNTIFS, etiquetas exactas de ocupación y búsqueda por subcadenas de adjetivos después de normalización manual. Los umbrales de 15, 20 o 25 filtran la presentación; no son criterios estadísticos preregistrados. No se informan tests inferenciales, tamaños de efecto, intervalos o sensibilidad. Productos, prompts, tamaño de lote, CSV con punto y coma y decisiones de normalización estructuran los resultados. Las coocurrencias no revelan por sí solas su origen en los datos de entrenamiento.

Los talleres reunieron ocho voluntarios autoseleccionados: siete mujeres y un hombre; cuatro europeos, uno sudamericano, uno de Oriente Medio y dos asiáticos. Las tablas de edad y experiencia con LLM suman siete, aunque N=8, sin una nota explícita de missingness. Todos eligieron ChatGPT pese a ofrecerse cuatro interfaces. Reaccionaron a una persona construida desde su nombre, discutieron cómo juzgamos a otros, cocrearon personas y conversaron con ellas. Se reportan 27 personas cocreadas, pero sus tablas suman 27 edades, 25 ubicaciones y 26 géneros. El audio se transcribió de forma anonimizada y luego se borró; no se comparten transcripciones completas.

El análisis cualitativo combina codificación descriptiva, in vivo, conceptual y afectiva con un codebook teórico provisional, seguida de codificación focalizada y de patrones. Los tres autores codificaron un subconjunto inicial con 92 % de acuerdo y kappa 0,48, y otro con 98 % y kappa 0,85; la primera autora codificó el resto. No se dan tamaños, unidades, prevalencias, tablas ni cálculos. Los participantes expresaron frustración, diversión o incomodidad, cuestionaron la neutralidad del modelo e identificaron asociaciones que los agregados podían ocultar. El anexo público contiene 18 personas de debate, ocho seleccionadas y diez residuales, y deja ver que incluso una búsqueda explícita de diversidad puede producir arquetipos tipificados.

La evidencia respalda una conclusión acotada: en estos lotes de finales de 2024 y comienzos de 2025, las salidas fueron mayoritariamente binarias y asociaron género con trabajo, ambición y personalidad. Hacer explícitos identidad o pronombres amplió etiquetas en algunos casos, sobre todo ChatGPT, pero no eliminó estereotipos. No demuestra cuál modelo actual es menos sesgado, que 1.720 filas sean ensayos independientes, que el prompting inclusivo resuelva el problema, que las coocurrencias tengan una causa identificada ni que ocho participantes representen una población amplia.

La reproducibilidad descriptiva es mejor de lo habitual: se recuperaron el PDF final abierto, tres XLSX con prompts, filas, fórmulas y tablas, y el anexo de once páginas. No hay hojas ocultas, nombres definidos ni celdas de error. Faltan snapshots exactos, parámetros, seeds, chats crudos, IDs, timestamps por petición, entorno ejecutable y trazabilidad explícita por fila. Tampoco se recuperaron dos anexos nombrados en el artículo, transcripciones, workbook de codificación o cálculos de fiabilidad. La lectura responsable usa estas personas para interrogar sesgos, nunca como sustitutos directos de investigación con personas reales.

Research question

How do synthetic persons generated by LLMs perform and reproduce gender in their labels, occupations, ambitions, and traits, and what additional biases does a participatory evaluation detect compared with an exclusively expert evaluation?

Method

Mixed design with three descriptive queries to ChatGPT 4o, Gemini 1.5 Flash, Cohere Command, and Pi.ai (1,720 rows generated in 68 batch requests), plus a pilot and two participatory workshops. Spreadsheets tally gender, age, occupations, and traits through formulas and manual normalization; the workshops are analyzed with two cycles of qualitative coding and partial agreement and kappa checks.

Sample: N=8 self-selected participants: seven women and one man; four Europeans, one South American, one from the Middle East, and two Asians. The age and LLM use duration tables report only seven cases. All chose ChatGPT for the activities.

Findings

  • The outputs were mostly binary and concentrated on young adults.
  • Female persons were more associated with creativity, freelance work, care, warmth, and empathy; male persons with technical or stable roles, ambition, analysis, and professional impact.
  • Making gender identity or pronouns explicit expanded labels in some batches, especially ChatGPT, without eliminating stereotyped associations.
  • In the ambiguous names query, Gemini remained almost entirely binary and Cohere only expanded in a limited way.
  • The two initial conditions of Pi in query 3 share 249 of 250 cells, so they are not independent realizations.
  • The workshops identified biases, discomfort, and assumptions of neutrality that aggregate counts could overlook.
  • Synthetic persons are more defensible as critical provocations than as substitutes for real users.
  • The three spreadsheets allow auditing prompts, rows, and descriptive counts better than isolated text.

Limitations

  • 1,720 rows come from only 68 batch requests; rows within each response are not independent repetitions.
  • There are no replicates per condition, seeds, variance across runs, inference, effect sizes, or intervals.
  • Web interfaces and model versions are not fixed; parameters, metadata, and raw chats are missing.
  • Prompts, products, batch size, and CSV format can structure the patterns.
  • Occupations and adjectives are standardized manually without a complete row-by-row transformation ledger.
  • Frequency thresholds are presentation filters, not preregistered statistical criteria.
  • The Demografix API uses a binary probability and does not demonstrate cultural neutrality of names.
  • Participatory sample N=8, seven women, self-selected and limited to ChatGPT during the workshops.
  • Participant and co-created person demographic tables contain incomplete denominators without explicit missingness notes.
  • Qualitative agreement does not publish subset sizes, units, prevalences, tables, or calculations.
  • The feminist lens is intentional and useful, but not a universal or neutral classification of bias.
  • The literature review is flexible and does not constitute a preregistered systematic review.
  • Two named annexes, transcripts, codebook, coding workbook, or executable environment were not recovered.
  • Co-occurrences do not identify causes in the training data or generalize to current versions.

What the study does not establish

  • It does not demonstrate that the 1,720 persons are 1,720 independent trials.
  • It does not demonstrate that one model is stably less biased than another.
  • It does not demonstrate that inclusive prompts eliminate gender stereotypes.
  • It does not demonstrate the cause of the patterns in the training data.
  • It does not demonstrate generalization from the workshops to a broad or gender-balanced population.
  • It does not demonstrate that synthetic persons accurately represent real users.
  • It does not validate using synthetic persons as a substitute for participatory research.
  • It does not allow exactly repeating the behavior of the live model or the complete qualitative analysis.

Traceability

Scope: Full text

Version: International Journal of Human-Computer Studies 205 (2025), article 103651, publisher version of record; three XLSX inquiry supplements and one PDF persona supplement

Consulted source: https://vbn.aau.dk/ws/files/802385111/1-s2.0-S1071581925002083-main.pdf

Review: Codex version-of-record full-text, 24-page visual, three-workbook formula-and-row, 11-page supplement visual, batch-independence, participatory-method, qualitative-validity and public-artifact audit, 2026-07-16

Approval: Codex fidelity pass, 2026-07-16

English translation: approved, 2026-07-18

Models evaluated

  • ChatGPT 4o (interfaz web)
  • Gemini 1.5 Flash (interfaz web)
  • Cohere Command (interfaz web)
  • Pi.ai (interfaz web)

Instruments and metrics

  • Consulta 1: campos Gender, Gender Identity y Pronouns
  • Consulta 2: once productos
  • Consulta 3: cincuenta nombres con probabilidad binaria Demografix entre 50 % y 70 %
  • Tres libros XLSX con prompts, filas, fórmulas COUNTIFS y tablas procesadas
  • Talleres W0, W1 y W2 con cuatro actividades participativas
  • Codificación cualitativa ecléctica, focalizada y de patrones
  • Anexo de personas para debate sobre derechos de género

Data used

  • 240 personas de la consulta 1
  • 880 personas de la consulta 2
  • 600 personas de la consulta 3
  • Lista Demografix de nombres con probabilidad binaria de género
  • 27 personas cocreadas en talleres
  • 18 personas del anexo de debate

Evidence and location

  • Metadata, theory, mixed design, workshops, results, limitations, and annexes: International Journal of Human-Computer Studies 205 (2025), article 103651, DOI 10.1016/j.ijhcs.2025.103651, 23 journal pages
  • Prompts, 1,720 rows, distributions, formulas, and descriptive transformations: Elsevier supplements mmc1.xlsx, mmc2.xlsx and mmc3.xlsx
  • Eighteen persons generated for the discussion panel: Elsevier supplement mmc4.pdf, 11 pages
  • Batch independence audit, Pi duplication, spreadsheets, participation, validity, and artifacts: reports/verification/article-235-gendering-participatory-batch-independence-spreadsheet-and-artifact-audit.json