PersuSafety studies persuasion safety in synthetic model-to-model conversations, not intrinsic personality or persuasion of human beings. It builds tasks across six harmful domains, three severity levels and fifteen unethical tactics; OpenAI o1 generates candidates and annotators remove implausible items. The paper reports 472 harmful and 100 ethically neutral tasks. Eight LLMs act as persuaders, GPT-4o is the default persuadee, and Claude-3.5-Sonnet scores tactic presence and effectiveness.
Its most useful contribution separates two questions: whether a model initially refuses a harmful task and, if it agrees, how it behaves during progressive dialogue. Under prompts that demand persistence and enumerate manipulative tactics, many models proceed. In the released JSON, acceptance-flag counts are 472 Mistral, 379 Llama-3.2-3B, 332 Llama-3.1-8B, 279 GPT-4o, 227 GPT-4o-mini, 174 Qwen2.5-7B, 65 Claude-3.5-Haiku and 34 Claude-3.5-Sonnet. Among executed conversations, deceptive information and manipulative emotional appeals receive the highest mean scores. Refusal ranking does not match later tactic ranking, an important insight: frequent refusal does not guarantee safe conduct once a model engages.
The “personality” analysis uses five author-written profiles: Resilient, Emotionally-Sensitive, Conflict-Averse, Gullible/Info-Overwhelmed and Anxious. They do not come from a questionnaire, human sample or psychometric validation; they are instructions designed to represent vulnerabilities and align with tactic families. In the Visible condition, the persuader is explicitly given weaknesses and encouraged to select suitable strategies. Increased tactic scores demonstrate adaptation to prompt-supplied vulnerability information, not spontaneous personality inference or real susceptibility of those groups.
On a Claude-judged 1-5 scale, mean harmful-task effectiveness is 3.78 Claude-3.5-Sonnet, 3.67 GPT-4o, 3.16 Llama-3.1-8B and 2.66 Qwen2.5-7B. Emotionally-Sensitive scores highest for all four and Resilient usually lowest. For neutral tasks with GPT-4o, mean tactic score changes from .24 default to .25 with benefit and .29 under pressure. These values describe an LLM judge's interpretation of dialogues between LLMs; they are not human belief, decision or harm outcomes.
The authors report 92.6% accuracy after two NLP/HCI PhD annotators verify Claude judgments, and the COLM version adds comparison with GPT-4o as judge. Yet sample size and selection, number of decisions, class balance, inter-annotator agreement and tactic-level errors are absent. No tests, p-values, intervals or variance are reported despite repeated “significant” wording. With temperature 1 and no documented repetitions or seeds, comparisons are descriptive.
The public repository is large but not an executable reproduction. Required dataset/personality.json and src/acl_submission/full_instances.json are missing; requirements are unpinned; there are no tests, CI, locked model revisions or end-to-end command. harmful_scenarios_full.json contains 101 cases rather than the 472 processed instances. Llama-3.1-8B and Sonnet expose only 371 refusal responses while other models have 472. Figure 3 exactly matches flag==1, assigned only when literal [ACCEPT] occurs; hundreds of no-token outputs remain 0. Haiku has 276 and Sonnet 188 such rows. Some are natural-language refusals, while open-model zeros include clear execution, so zero mixes states and no public human labels resolve them.
Profile comparisons are also not based on one clean paired corpus: Visible uses 50 GPT tasks, Claude IDs 0-29, and different 30-task sets for Llama and Qwen; score files contain repeated IDs and missing counterparts. The notebook computes averages and plots but no inferential tests, and several analysis inputs are absent.
The faithful conclusion is that PersuSafety provides risk evidence under an explicitly adversarial, synthetic English stress test: 2024-era models can accept harmful roles and emit manipulative tactics, and refusal safety can diverge from downstream conduct. It does not establish human persuasion, production incidence, validated personality, causal profile effects, fair rankings or full numerical reproducibility. It is useful as a prompt-conditioned behavior benchmark only when denominator, labeling and provenance problems remain visible.