Disentangling Intent from Role: Adversarial Self-Play for Persona-Invariant Safety Alignment

Trait induction and control2026arXivApproved editorial review

Authors: Jiajia Li, Xiaoyu Wen, Zhongtian Ma, Shuyue Hu, Qiaosheng Zhang, Zhen Wang

Keywords: Persona conditioning, Safety and bias

Source: Open primary source (opens in a new tab)

6
Authors
8
Findings
20
Limitations
4
Evidence

Editorial summary

English

PIA combines two components: Persona Lineage Evolution searches for role descriptions that break safety refusals, and Persona-Invariant Consistency Learning trains the persona-conditioned model toward its persona-free output distribution alongside DPO and SFT. In the reported tables, PLE outperforms Persona-GA and PICL sharply lowers mean attack success under unseen personas, from .601 to .054 for Qwen2.5-7B and from .302 to .052 for Llama-3.1-8B. This is meaningful behavioral-defense evidence within the paper's protocol, but it does not demonstrate the claimed structural decoupling. The theoretical bound is derived for KL(persona model || reference), whereas the method reverses the arguments. The code further masks to the teacher's top 100 tokens without renormalization, so the partial sum can be negative and is not a KL divergence. WildGuard filters data, drives evolution and judges every safety result, with no human validation or second judge, and each target response is sampled once. SafeRLHF-unsafe is not genuinely out of distribution: 3,608 of 5,270 prompts exactly match training prompts. The release omits the final training corpus, adapters, responses, judge labels and numeric results; its Hugging Face model repository contains only the paper. PICL also increases benign refusals and slightly reduces Qwen's mean capability. The work was accepted at ICML 2026, but the available evidence establishes lower WildGuard-labeled ASR, not mechanistic invariance or independently reproduced general robustness.

Español

PIA combina dos piezas: Persona Lineage Evolution busca descripciones de rol que hacen fallar las negativas de seguridad, y Persona-Invariant Consistency Learning entrena al modelo para acercar su distribución con persona a la respuesta sin persona, junto con DPO y SFT. En las tablas, PLE supera a Persona-GA y PICL reduce mucho el ASR medio con personas no vistas: de .601 a .054 en Qwen2.5-7B y de .302 a .052 en Llama-3.1-8B. El resultado es relevante como defensa conductual dentro de ese protocolo, pero no demuestra el anunciado desacoplamiento estructural. El límite teórico se deriva para KL(modelo con persona || referencia), mientras el método invierte los argumentos; además, el código enmascara los 100 tokens principales sin renormalizar, por lo que la suma puede ser negativa y no es una KL. WildGuard selecciona datos, guía la evolución y juzga todos los resultados, sin validación humana ni segundo juez, y sólo hay una generación estocástica por caso. SafeRLHF-unsafe tampoco es realmente OOD: 3.608 de 5.270 prompts coinciden exactamente con entrenamiento. La publicación no entrega el corpus final de entrenamiento, adaptadores, respuestas, etiquetas del juez ni resultados numéricos; el repositorio de modelo en Hugging Face sólo contiene el PDF. PICL también aumenta el rechazo benigno y reduce algo la media de capacidad de Qwen. El trabajo fue aceptado en ICML 2026, pero la evidencia disponible sustenta menor ASR según WildGuard, no invariancia mecanística ni robustez general reproducida independientemente.

Research question

Can a coevolution between adversarial personas and a defender model make refusal toward a harmful intent stable against the requested role, without destroying general utility or role-play capability?

Method

PLE evolves 35 elite personas over 40 generations via lineage graph, credit propagation, UCB, mutation, and crossover. Each generation uses 100 fixed JBB instructions and 50 sampled from PKU-SafeRLHF. PICL mixes 10,000 DPO pairs with persona, 10,000 DPO without persona, and 15,000 SFT examples; it adds a consistency between logits of the no-persona condition, stopped as teacher, and the with-persona condition. The review inspected the 21 pages, TeX, code and history, published datasets, Hugging Face artifacts, OpenReview/ICML metadata, formulas, training configuration, overlaps, and parsers.

Sample: The published harmful pool contains 20,430 unique training prompts. The article declares 35,000 examples for PICL, but that final corpus is not published. The evaluation aggregates multiple benchmarks with one generation per prompt and temperature .7. SafeRLHF-unsafe has 5,270 rows, 3,608 exactly overlapping with training; DAN retains 857 logical objects but 826 unique prompts, and HarmBench 400 rows but 393 unique prompts. Two JSONL files contain objects split by unescaped line breaks, and line-by-line loaders lose records.

Findings

  • Mean ASR with OOD personas drops from .601 to .054 for Qwen and from .302 to .052 for Llama across five published benchmarks.
  • Mean ASR against direct harm drops from .244 to .115 for Qwen and from .102 to .036 for Llama.
  • Mean benign rejection rises from .054 to .091 in Qwen and from .128 to .179 in Llama.
  • Mean Qwen capability drops from .650 to .629, with ARC from .669 to .593; Llama remains practically the same, .601 to .600.
  • PLE outperforms Persona-GA across the nine columns of the Qwen table, but the same WildGuard is used to optimize and evaluate.
  • The implemented KL direction is not the derived variational bound and the unrenormalized top-100 can produce a negative loss.
  • SafeRLHF-unsafe has 68.46% exact row overlap with the published training pool.
  • The accepted OpenReview version lists five authors; arXiv v1 and the repository list six.

Limitations

  • WildGuard filters data, builds part of the training, guides PLE fitness, and labels all safety and overrefusal results.
  • There is no human annotation, independent judge, calibration, agreement, or judge error analysis.
  • A single response per case is generated at temperature .7; there are no repetitions, intervals, or statistical tests.
  • The KL inversion is justified by coverage, but it ceases to optimize the derived mutual information upper bound.
  • The top-100 mask does not renormalize distributions and its partial sum is neither a KL divergence nor does it guarantee non-negativity.
  • The no-persona teacher is not always safe; the base Qwen reaches direct ASR .329 on DAN and .458 on WildJailbreak-harm.
  • PIC is computed via teacher forcing over tokens of chosen completions, not over full distributions of generated sequences.
  • Mutual information, internal representation, causality, or structural independence are not measured.
  • SafeRLHF-unsafe is contaminated in 3,608/5,270 prompts and does not support an OOD evaluation.
  • DAN and HarmBench contain duplicates; StrongREJECT and MaliciousInstruct share 12 prompts.
  • Baseline elite contains 35 rows but only 33 unique personas.
  • The paper declares 5k SafeRLHF-unsafe and 1.405 DAN; the repo contains 5,270 and 857 logical objects.
  • PKU-SafeRLHF-Train-unsafe and DAN are not strictly valid JSONL; the published parsers omit broken lines.
  • The paper declares global batch 64 and 546 steps; run.sh configures 3x1x14=42 and approximately 833 steps for 35k rows.
  • No training.jsonl, adapters, checkpoints, responses, judgments, logs, or recomputable tables are published.
  • The Hugging Face repository labeled as model contains only README and PDF.
  • Several README paths do not exist and end-to-end execution does not follow the instructions as written.
  • TRL and IFEval remain without version/commit; there is no lockfile, container, CI, or tests.
  • Exact reviews of models and the versioned identity of Qwen3-Max and WildGuard are missing.
  • Releasing high-ASR adversarial personas introduces dual-use risk without a tiered access process.

What the study does not establish

  • That safety is structurally or mechanistically decoupled from the persona.
  • That the presented mutual information upper bound has been minimized.
  • That the implemented top-100 loss is a valid KL.
  • That the improvement transfers to human judges or independent classifiers.
  • That SafeRLHF-unsafe is an OOD set.
  • That robustness persists against adaptive attacks not optimized for WildGuard.
  • That there is no overrefusal or capability cost.
  • That all forms of benign role-play outside CharacterEval and RoleBench are preserved.
  • That the results are reproducible from the public artifacts.
  • That ICML acceptance resolves the data, code, judge, and construct limitations.

Traceability

Scope: Full text

Version: arXiv:2605.01899v1; 21-page PDF, TeX, official repository commit e3ce359, Hugging Face artifacts and ICML/OpenReview metadata audited

Consulted source: https://arxiv.org/abs/2605.01899v1

Review: Codex 21-page visual full-text, TeX, official repository, Hugging Face, theory, judge, leakage, data, code and reproducibility audit, 2026-07-17

Approval: Codex fidelity pass, 2026-07-17

English translation: approved, 2026-07-18

Models evaluated

  • Qwen2.5-7B-Instruct
  • Llama-3.1-8B-Instruct
  • Qwen3-Max as mutation/crossover generator
  • WildGuard-7B as sole safety judge
  • Self-RedTeam, SmoothLLM, LLM self-eval, SFT and DPO baselines

Instruments and metrics

  • Attack Success Rate labeled by WildGuard
  • Benign Refusal-to-Answer rate labeled by WildGuard
  • IFEval, AI2-ARC, GPQA-diamond and MMLU
  • CharacterEval and RoleBench
  • BGE-M3 persona similarity
  • Persona Lineage Evolution and Persona-GA

Data used

  • JBB-Behaviors harmful and benign
  • PKU-SafeRLHF train/test-derived harmful and safe subsets
  • StrongREJECT, WildGuardTest, XSTest, AdvBench, DAN and HarmBench
  • MaliciousInstruct, OR-Bench, WildJailbreak and TrustLLM
  • Databricks Dolly 15k
  • 35 released attack elite, 35 baseline elite and 35 OOD elite persona rows

Evidence and location

  • Text, equations, results, appendices, prompts, and configuration: arXiv:2605.01899v1; PDF sha256 843b34676959dc5ba81db918146f96bc872c6cb7166ac5c48d0c30057f7ba11d; TeX sha256 6ede9d3a3f86736b2117bf297224c0e614f3df74896800431d8f5197f95e8bfd
  • Code, data, JSONL, configuration, and absent artifacts: JiajiaLi-1130/PIA commit e3ce35985eee73b0ddaf906facdb0f6dc18f4b81; archive sha256 18c5fdbaee3a476c44606d014cd40e26404e4642d523e1555a83c03d0b75412b
  • ICML 2026 acceptance and authorship/version derivation: OpenReview pTz80QaxqD and ICML 2026 poster 61505; snapshots cached under .cache/editorial-sources/article-349/supplements/
  • Audit of theory, judge, leakage, data, code, and reproducibility: reports/verification/article-349-pia-theory-judge-leakage-data-code-and-reproducibility-audit.json