When Does Personality Composition Matter for Multi-Agent LLM Teams?

Society, culture, and collective behavior2026arXivApproved editorial review

Authors: Aryan Keluskar, Amrita Bhattacharjee, Huan Liu

Keywords: Personality, Persona conditioning, Multi-agent systems, Big Five, Agreeableness, LLM-as-judge, Reproducibility

Source: Open primary source (opens in a new tab)

3
Authors
10
Findings
18
Limitations
7
Evidence

Editorial summary

English

This COLM 2026 paper studies how personality prompts change communication and outcomes in multi-agent LLM teams. The primary intervention prepends seven Goldberg low- or high-Agreeableness adjectives to every agent. Low-A is intensely valenced, very unkind, uncooperative, selfish, distrustful, cold, harsh, and unsympathetic, so it operationalizes an instructed persona rather than a measured psychological personality. The study compares three MultiAgentBench domains: three-agent collaborative coding, five-agent open-ended research ideation, and two-agent buyer-seller bargaining. GPT-4o-mini labels segments as questions, disagreements, suggestions, and acknowledgments. Phi groups the first three act counts against acknowledgments. Loaded low-A raises phi to roughly 0.93-0.95: Claude, GPT-4o, and Grok become disagreement-dominated, whereas DeepSeek shifts mainly through suggestions. This is an LLM-labeled act-composition measure, not literally a percentage of messages or a direct measure of communication quality, divergence, or successful convergence. In coding, LLM-judged planning declines but milestone counts remain near baseline for Claude, GPT-4o, and DeepSeek. Grok falls from 14.4 to 10.9 and is the only declared exception, but the inference is fragile: on the five shared tasks, a paired t-test gives p=0.013 and Wilcoxon p=0.0625; the prose reports p=0.017 while the table marks it with a star defined as p<0.01. In research, the main direction survives task-matched comparison for the three models with released logs: GPT-4o drops 6.93 milestones across 15 tasks, Grok 6.17 across 9, and DeepSeek 3.87 across 22. However, actual samples contradict the stated n=30 per condition: Grok has 15 baseline tasks versus 9 low-A tasks, and DeepSeek 25 versus 22, with unequal repetitions. Paper means use these unmatched collections, and the blanket p<0.0001 claim becomes about 0.0007 for Grok and 0.0014 for DeepSeek when task is respected as the unit. In bargaining, released data show agreement moving from 37% to 1% for GPT-4o, 18% to 0% for DeepSeek, and 31% to 0% for Claude; high-A raises them to 71%, 26%, and 60%. A never-accept control suggests low-A is not always a flat refusal because some agents keep moving offers while rarely accepting. Still, Table 2 says Claude baseline is 40% rather than the 31% in prose, Table 3, and data, and Table 3 prints [0,0] for Claude 0/100 when a Wilson interval is about [0,4]. The strongest methodological contribution of v2 is the neutral paraphrase control: direct, candid, independent-minded, skeptical of consensus, and efficiency-oriented. It removes cross-model convergence and largely removes disagreements, showing that much of the Goldberg effect comes from hostile wording. Residual outcomes are heterogeneous: GPT-4o research largely recovers, Grok and DeepSeek remain lower, Claude improves in research, and neutral coding can be worse than loaded low-A in some cells. There is therefore no uniformly directional residual or psychometrically equivalent manipulation. Artifact-mediated buffering is a plausible hypothesis but is not causally identified. Domains differ in agent count, tasks, objectives, actions, model coverage, sample sizes, and judges, and the competitive/high-structure cell is absent. Format extraction and LLM code ratings do not demonstrate that syntax and semantics filtered communication degradation; no comparable functional execution suite is reported. The public artifact is valuable: Apache-2.0 licensing, 2,679 JSONL logs, about 293 MB of results, caches, and scripts. Offline reconstruction reproduces six CSVs byte-for-byte and the seventh except for floating-point last digits. But Claude research logs, Grok bargaining logs, and Kimi validation are absent; the README has no reproduction commands; secondary statistical scripts use stale paths; dependencies are unpinned; there are no study tests or CI; and the runner sets the evaluated model as judge while the appendix says GPT-4o judged code. The defensible conclusion is that low-A persona wording strongly changes labeled communication acts and can harm some multi-agent outcomes, especially research and bargaining acceptance. The paper contributes an important prompt-valence warning and a rich log release, but it does not establish stable personality, a causal artifact-structure mechanism, or a deployment-ready design rule.

Español

Este trabajo aceptado en COLM 2026 estudia cómo prompts de personalidad alteran la comunicación y los resultados de equipos multiagente de LLM. La intervención principal antepone a todos los agentes siete adjetivos Goldberg de baja o alta Amabilidad. La condición low-A usa una formulación intensamente negativa, muy poco amable, no cooperativa, egoísta, desconfiada, fría, dura y poco comprensiva, por lo que operacionaliza una persona instruida y cargada de valencia, no una personalidad psicológica medida. El estudio compara tres dominios de MultiAgentBench: programación colaborativa de tres agentes, generación abierta de ideas de investigación con cinco agentes y negociación comprador-vendedor con dos agentes. GPT-4o-mini clasifica segmentos en preguntas, desacuerdos, sugerencias y reconocimientos. El indicador phi agrupa los tres primeros actos frente al reconocimiento. Low-A eleva phi aproximadamente a 0,93-0,95: Claude, GPT-4o y Grok se desplazan hacia desacuerdos, mientras DeepSeek lo hace sobre todo mediante sugerencias. Este resultado describe la composición de actos etiquetados por un juez LLM; no equivale literalmente al porcentaje de mensajes ni mide por sí solo calidad, divergencia o convergencia exitosa. En programación, la planificación juzgada por LLM empeora, pero los hitos permanecen cerca del baseline en Claude, GPT-4o y DeepSeek. Grok cae de 14,4 a 10,9 y es la única excepción declarada, aunque su inferencia es frágil: sobre las cinco tareas compartidas, una t emparejada da p=0,013 y Wilcoxon p=0,0625; el texto publica p=0,017 mientras la tabla lo marca con un asterisco definido como p<0,01. En investigación, las caídas principales sí sobreviven una comparación emparejada por tarea en los tres modelos cuyos logs están publicados: GPT-4o baja 6,93 hitos en 15 tareas, Grok 6,17 en 9 y DeepSeek 3,87 en 22. Sin embargo, los tamaños reales contradicen n=30 por condición: Grok usa 15 tareas base frente a 9 low-A y DeepSeek 25 frente a 22, con réplicas desiguales. Las medias del paper usan esas colecciones no equivalentes y los p<0,0001 uniformes se reducen aproximadamente a 0,0007 y 0,0014 para Grok y DeepSeek al respetar la unidad tarea. En negociación, los datos liberados muestran que el acuerdo pasa de 37% a 1% en GPT-4o, de 18% a 0% en DeepSeek y de 31% a 0% en Claude. High-A sube a 71%, 26% y 60%. El control never-accept muestra que low-A no se limita siempre a una negativa plana: algunos agentes siguen moviendo ofertas pero rara vez aceptan. Aun así, la Tabla 2 dice 40% base para Claude frente al 31% del texto, Tabla 3 y datos; además imprime un intervalo [0,0] para 0/100 acuerdos Claude, cuando el Wilson correcto es aproximadamente [0,4]. La aportación metodológica más sólida de la v2 es el control de paráfrasis neutral: «directo, franco, independiente, escéptico del consenso y orientado a eficiencia» elimina la convergencia entre modelos y reduce los desacuerdos, mostrando que buena parte del efecto Goldberg procede de adjetivos hostiles. Los resultados restantes son heterogéneos: GPT-4o recupera buena parte de investigación, Grok y DeepSeek siguen por debajo, Claude mejora en investigación y la programación neutral incluso empeora más que low-A en algunas celdas. Por ello no hay un efecto residual uniformemente direccional ni una manipulación psicométricamente equivalente. La explicación de «amortiguación mediada por el artefacto» es razonable como hipótesis, pero no está identificada causalmente. Los dominios difieren en número de agentes, tareas, objetivos, acciones, modelos, tamaños y jueces, y falta la celda competitiva de alta estructura. Que el código se extraiga con formato o reciba una puntuación LLM no demuestra que restricciones sintácticas y semánticas hayan filtrado el deterioro comunicativo; no se presenta una batería de ejecución funcional comparable. El artefacto público es valioso: licencia Apache-2.0, 2.679 logs JSONL, unos 293 MB de resultados, caches y scripts. La reconstrucción offline reproduce seis CSV byte a byte y el séptimo salvo redondeo de coma flotante. Pero no contiene los logs de investigación Claude, negociación Grok ni la validación Kimi; el README no da comandos de reproducción; scripts estadísticos secundarios conservan rutas obsoletas; las dependencias no están fijadas; no hay tests o CI del estudio; y el runner configura como evaluador al propio modelo, mientras el apéndice afirma que GPT-4o juzga el código. La conclusión defendible es que la redacción de una persona low-A cambia mucho los actos comunicativos etiquetados y puede perjudicar ciertos resultados multiagente, especialmente investigación y aceptación en negociación. El trabajo aporta una advertencia importante sobre valencia del prompt y un conjunto de logs rico, pero no demuestra personalidad estable, un mecanismo causal de estructura del artefacto ni una regla lista para despliegue.

Research question

Do Agreeableness prompts change the communication and performance of LLM multiagent teams differently depending on the structure of the outcome and the alignment of objectives, and how much of the effect is due to the negative valence of the adjectives?

Method

Big Five prompting experiments in MultiAgentBench with homogeneous low-A, high-A and baseline teams, Openness and Conscientiousness ablations, neutral paraphrase, a one-challenger pilot and never-accept control. Compares programming, research and negotiation; classifies acts with GPT-4o-mini and evaluates milestones, planning, code quality, agreements and offer movement through judges and automatic logs.

Sample: Programming: four models, five shared tasks for the main comparisons and 10-15 runs per condition, although GPT-4o, Grok and DeepSeek publish 20 baselines over ten tasks. Research: the paper declares 15 tasks and n=30 per condition; GPT-4o complies, Grok publishes 35/23/26 base/low-A/high-A runs over 15/9/11 tasks and DeepSeek 55/49/52 over 25/22/24; Claude logs are not published. Negotiation: 50 tasks and 100 runs per condition for GPT-4o, DeepSeek and Claude; the cited Grok logs are not published. Neutral controls: normally ten runs over five tasks per model and domain. The heterogeneous pilot covers positions 0-2 in GPT-4o, Grok and DeepSeek.

Findings

  • The Goldberg low-A prompt raises phi to approximately 0.93-0.95 across the four models.
  • Claude, GPT-4o and Grok reach high phi through disagreements; DeepSeek does so mainly through suggestions.
  • The neutral paraphrase eliminates convergence between models and almost all disagreements, evidencing a strong prompt valence effect.
  • On the five shared programming tasks, Claude, GPT-4o and DeepSeek show no clear drop in milestones.
  • The Grok drop in programming is sensitive to the test and does not unambiguously meet the p<0.01 threshold indicated in the table.
  • The research drops survive task-level matching in GPT-4o, Grok and DeepSeek, although with less precision than published.
  • Low-A drastically reduces acceptance in negotiation for GPT-4o, DeepSeek and Claude; high-A increases it.
  • The never-accept control indicates that low-A can maintain counteroffers yet almost never finalize agreements.
  • The neutral effect on outcomes is heterogeneous by model and domain, not uniformly attenuated or directional.
  • The unified repository analysis regenerates the published CSVs from the released logs.

Limitations

  • Low-A uses hostile adjectives and does not constitute a clean manipulation or a measurement of psychological Agreeableness.
  • The neutral paraphrase simultaneously changes valence, intensity and content; it diagnoses confounding, but does not identify a pure residual trait.
  • Phi counts multi-label acts and is misinterpreted in parts of the text as a fraction of messages.
  • Phi does not directly measure quality, correct convergence, proposal integration or performance.
  • The Kimi validation omits the suggestion category and no labels, code or sampling of the second judge are published.
  • The appendix claims correlations >0.80 for baseline/high-A but its table shows Claude baseline r=-0.08.
  • The Grok and DeepSeek research results mix different task sets and numbers of replicates between conditions.
  • Execution-level tests and bootstraps incur pseudoreplication by ignoring clustering by task.
  • Multiplicity across models, traits, directions, domains, metrics and controls is not corrected globally and consistently.
  • The tables mix baselines of five and ten tasks; the DeepSeek value 12.06 in Table 5 does not reproduce as all-task or matched mean.
  • The Grok significance in code is labeled p<0.01 even though the text itself gives p=0.017.
  • The Claude 0/100 negotiation interval is misprinted as [0,0] and Table 2 uses 40% base versus the 31% reproducible.
  • Domains differ on many variables and one cell of the 2x2 design is missing, so the structure of the artifact is not causally identified.
  • Milestones and code quality are LLM judgments; there is no functional validation executed comparable across domains.
  • The code configures self-evaluation by model, in conflict with the claim that GPT-4o judges all code quality.
  • Claude research results, Grok negotiation results and the Kimi validation are not published.
  • README, statistical paths, dependencies, seeds, model snapshots, tests and CI do not allow stable end-to-end reproduction.
  • The challenger pilot is small and does not cover sufficient compositions, roles or domains for a general recommendation.

What the study does not establish

  • It does not establish that LLMs possess stable or comparable personalities with human traits.
  • It does not causally isolate Agreeableness from hostility, toxicity or instruction following.
  • It does not demonstrate that the formal structure of the artifact causes programming robustness.
  • It does not demonstrate that phi is a validated measure of team quality or convergence.
  • It does not prove that the produced code works better or equally through real tests.
  • It does not support p<0.0001 for all research effects when task is the unit of inference.
  • It does not establish that a low-A challenger in the leader position is a general optimal rule.
  • It does not demonstrate uniform, stable or model-independent neutral effects.
  • It does not offer complete reproduction of all tables of the accepted paper.
  • It does not validate deployment in real teams, organizations, persistent agents or systems with sensitive tools.

Traceability

Scope: Full text

Version: arXiv:2606.27443v2; accepted at COLM 2026

Consulted source: https://arxiv.org/abs/2606.27443v2

Review: Codex 20-page full-text visual, v2 text, prompt, metric, task-matched statistical, code, artifact, reproducibility, ethics and claim audit, 2026-07-16

Approval: Codex fidelity pass, 2026-07-16

English translation: approved, 2026-07-18

Models evaluated

  • Claude Sonnet 4
  • GPT-4o
  • Grok-3
  • DeepSeek V3.1
  • GPT-4o-mini as communication-act judge
  • Kimi K2.6 as unreleased second judge

Instruments and metrics

  • Goldberg bipolar Big Five adjective prompts
  • Nine-level linguistic qualifier protocol
  • Neutral low-cooperation paraphrase
  • MultiAgentBench milestone rubric
  • Multi-label communication-act classifier
  • Communication state phi
  • Mechanism decomposition ratio delta
  • LLM-judged planning and code-quality scales
  • Bargaining agreement and offer-movement logs
  • Single-challenger position sweep
  • Never-accept bargaining control

Data used

  • MultiAgentBench coding tasks
  • MultiAgentBench research tasks
  • MultiAgentBench bargaining tasks
  • Released MARBLE experiment JSONL logs
  • Released GPT-4o-mini classifier cache

Evidence and location

  • Metadata, COLM 2026 acceptance, version and extension: Official arXiv record and arXiv:2606.27443v2, checked 2026-07-16
  • Prompts, domains, models and communication phi: arXiv v2, Sections 3.1-3.3 and Figure 5
  • Programming, research and negotiation results: arXiv v2, Sections 4.1-4.2 and Tables 1-3
  • Neutral paraphrase and outcome heterogeneity: arXiv v2, Section 4.3 and Tables 4-5
  • Judges, effects, ablations, robustness and examples: arXiv v2, Appendices A-L
  • Actual sizes, task sets, CSVs and code discrepancies: Public repository aryankeluskar/colm2026-multi-agent-llm-teams at commit c33fc25bcadb36bb54d4123c8f2021bf90deae2a
  • Task-matched reanalysis and consolidated audit: reports/verification/article-293-colm-task-set-pseudoreplication-prompt-valence-judge-artifact-and-claim-audit.json