The paper asks whether adding simulated demographic identities to a prompt changes labels and rationales produced by three LLMs on tasks with different levels of subjectivity. This review uses the definitive EACL 2026 version rather than only the preprint: a 19-page long paper, pages 1152–1170, DOI 10.18653/v1/2026.eacl-long.52, together with the public data, results, and code at commit 71c2bc22c08d052de70399e955197104c3d428f8.
The design has two branches. For HateXplain, the authors select 500 posts: 301 Normal, 75 Offensive language, and 124 Hate speech. Cases with three different labels and non-normal cases with fewer than three rationales are excluded. Every post is processed with 21 single-attribute personas: ages 15/35/65; male/female gender; no formal/high school/higher education; White/Black/Asian race; Christian/Muslim/Jewish/Atheist/Hindu religion; Left/Right/Centrist politics; and Not/Somewhat lonely. CoS-E uses 500 questions and SST-2 uses 263 sentences, both from BRWRR. These tasks use 12 composite personas: age 25/45 × male/female × African American/Hispanic/Caucasian, compared with six human annotation groups defined by age and ethnicity.
The evaluated systems are Mistral-Medium-3.1, Qwen3-32B, and GPT-OSS-120B through OpenRouter. Every persona and baseline is run three times. Metrics are Macro-F1 for HateXplain and SST-2, accuracy for CoS-E, MAE and mean error for HateXplain severity, Token-F1 and IOU-F1 for rationales, and Krippendorff's alpha for inter-persona agreement. Rationales are word lists requested from the model and subsequently mapped to binary masks over the input; they are not attribution measurements from internal activations.
The planned size per model is 31,500 persona completions for HateXplain, 18,000 for CoS-E, and 9,468 for SST-2, plus 3,789 baselines: 62,757 per model and 188,271 overall. A complete audit of all 27 released result directories finds 188,212 rows, 59 short. Seventeen persona responses are missing: 14 Mistral HateXplain, two Qwen HateXplain, and one GPT-OSS SST-2. In addition, Qwen's run-3 HateXplain baseline stops at s0457 and omits the contiguous tail s0458–s0499, 42 examples. The label CSV computes this baseline on 458 cases without reporting n; the non-normal rationale CSV evaluates 187 of 199 and does report 12 missing. Among present rows, there are no invalid JSON lines, duplicates, or HateXplain labels outside the allowed vocabulary.
On HateXplain, classification effects are strongly model-dependent. The paper reports significant improvement over baseline for 11 of 21 Mistral personas and significant degradation for 16 of 21 Qwen personas. GPT-OSS changes less. No model-persona pair significantly improves rationale quality, while several degrade it. The abstract's broad statement that persona prompting improves hate-speech classification therefore needs qualification: it primarily describes Mistral, not a stable cross-model effect.
All three models over-classify severity. GPT-OSS often maps Offensive to Hate, Mistral has high Normal-to-Offensive and Offensive-to-Hate rates, and Qwen is less likely to predict Hate but still shifts many Normal examples to Offensive. Left-wing personas predict Hate more often than Right or Centrist personas; the quantified Mistral example is 53%, 32%, and 24%. These findings show that demographic wording can change moderation decisions; they do not show that human political groups behave this way.
Inter-persona agreement is often high, but it is not equivalent to an absence of steering. HateXplain label alpha by group ranges roughly .81–.86 for GPT-OSS, .57–.89 for Mistral, and .52–.66 for Qwen. CoS-E label/rationale alpha is .93/.80, .95/.85, and .88/.63; SST-2 is .96/.72, .98/.89, and .84/.69. At the same time, Bonferroni-corrected Stuart–Maxwell tests find many distributional shifts, and within-attribute disagreement reaches 56.4% for Qwen political personas. High alpha can coexist with systematic shifts on a substantial minority of samples.
On CoS-E and SST-2, GPT-OSS and Mistral show no clear statistical benefit over baseline. Qwen degrades, including a 9.2-point drop for the Black Young Female persona on SST-2 and a rationale loss for Black Old Male. Personas that best match a group's annotations are usually not the demographically namesake personas. All models consistently perform better against annotations from older and White/African American groups than against several young or Hispanic groups; prompting does not remove that pattern.
The demographic-alignment interpretation has a central confound. The baseline removes all persona context, while the treatment adds much more than an age, gender, race, religion, or political label. It instructs the model to “step into the shoes,” imagine a whole life shaped by the attribute, let background, beliefs, and life experiences guide judgment, and remain in character. There is no control with the same role-play instruction and length but no attribute, no counterbalanced paraphrases, fictional identities, irrelevant attributes, or demographic-label-only treatment. Differences therefore do not isolate identity from learned stereotype, role compliance, prompt length, and chain-of-thought.
The prompt also assumes that one demographic attribute homogeneously shapes thoughts, emotions, sensitivities, and beliefs. That wording can induce the very stereotypes later observed in qualitative reasoning, especially for education, race, and politics. It is useful as a probe of model associations, but not as a validated simulation of a real person or an estimate of a group's preferences. The six BRWRR groups do not validate generated reasoning either; they only permit comparison of labels and selected tokens with age/ethnicity group aggregates.
Rationale measurement has further limitations. The code whitespace-tokenizes input_text and searches for every generated phrase; if the complete phrase is absent, it accepts the first longest matching subsegment. If no match exists, it produces zeros. Although compute_rationale_mask returns unmatched phrases, compute_rationale_binary discards that list and the result JSON does not preserve a failure flag. Across 188,212 rows, there are 513,855 non-empty segments: 504,930 fully match, 1,455 match only partially, and 7,470 do not match. There are 1,204 rows with no rationale list. Among the 199 non-normal HateXplain examples actually used for rationale evaluation, 83 responses have an empty rationale list and 176 masks are all-zero. Metrics include those masks, but the paper does not report this failure rate or separate semantic selection failure from extraction failure.
The matcher can also reward one partial word from a hallucinated phrase and marks only one occurrence when a term repeats. It does not record which portion was discarded. On the positive side, released mask lengths match input token counts and the six BRWRR group masks, so there is no detected structural length misalignment. The limitation concerns validity and traceability of matching rather than matrix shape.
Confidence intervals are described as sample-level bootstrap intervals with 1,000 iterations. The paper does not state a multiplicity correction for the many persona-versus-baseline performance comparisons; the documented Bonferroni procedure applies to within-attribute Stuart–Maxwell tests. Interpreting 11/21 or 16/21 findings at p<.05 without an explicit testing family can inflate false positives. More importantly, the repository contains no implementation of bootstrap, Stuart–Maxwell, p-values, or Bonferroni, although these analyses support the figures' significance markers and Table 11. Only final CSVs and plots are deposited, so the inferential results cannot be regenerated from the published code.
The result audit finds 12 Mistral CoS-E responses with an empty model_answer, five, three, and four by run. Accuracy scripts silently exclude them from the denominator, and CSV totals fall to 497–499 for affected personas. A HateXplain label of None would likewise be omitted. Excluding failures may overstate performance and, more importantly, can compare persona and baseline on different sample sets unless the bootstrap enforces a paired intersection. The paper does not specify how failed completions are handled statistically.
The public artifact contains datasets and 188,212 responses across 11,394 files, but it is not an executable reproduction. Seventeen of 25 analysis scripts set REPO_ROOT=THIS_DIR.parent.parent, which resolves from analysis/<dataset>/code to analysis rather than the repository root; they look for nonexistent analysis/results and analysis/datasets. All 18 inference scripts point to ../datasets/personas_&_questions or ../datasets/results_* paths that also do not match the published tree. Running persona_accuracy_per_run.py completes with no results after warning that all nine result directories are absent.
There is no requirements file, pyproject, lockfile, environment specification, test suite, CI, or license. The README lists scripts but provides no commands, Python version, installation instructions, exact order, or single pipeline. The paper says default sampling parameters, while code explicitly sets temperature=1.0 for GPT-OSS, .7 for Mistral, and .6 plus top_k=20, min_p=0, and top_p=.95 for Qwen; GPT-OSS uses medium reasoning effort. OpenRouter's effective provider, model snapshots, dates, seeds, and serving metadata are not pinned. The Mistral prompt also contains a missing comma between rationale and reasoning in its JSON example, although response_format=json_object is requested.
Scripts retry three times only for capacity/429 errors; other failures abandon the sample. API failures can therefore leave partial files, as the release demonstrates. Parse failures are stored as empty values and may subsequently be excluded. No automated check enforces cardinality, schema, coverage, common sample sets, or table regeneration before release.
The defensible contribution is a broad comparative probe showing, across three tasks and three models, that demographic prompts can shift labels without improving rationale correspondence or reliably matching namesake demographic annotations. It also exposes moderation over-flagging and stereotypes that are useful for auditing. It does not establish that LLMs reason like real people, that a demographic group shares the simulated perspective, that chain-of-thought is a faithful explanation, that persona prompting generally improves classification, or that high alpha proves causal resistance to steering. Missing statistical code and 59 absent completions mean significance claims must be treated as reported paper results that cannot be fully verified from the released artifact.