Chameleon LLMs: User Personas Influence Chatbot Personality Shifts

Evaluation and psychometric validity2025ACL AnthologyApproved editorial review

Authors: Jane Xing, Tianyi Niu, Shashank Srivastava

Keywords: Conversational Adaptation, Context Conditioning, Large Language Models, Big Five, IPIP 50-item, User Personas, Persona Simulation, Chameleon Effect, Long-context Prompting, Personality Measurement, WildChat, Alignment, Human-AI Interaction

Source: Open primary source (opens in a new tab)

3
Authors
12
Findings
17
Limitations
12
Evidence

Editorial summary

English

This study asks whether a user's persona changes how a chatbot expresses and reports Big Five traits after a conversation. GPT-4o mini enacts one of a nominal set of 100 extreme tropes, actually 99 unique strings because one is duplicated, and holds a 20-turn dialogue with seven chatbots: GPT-4o, GPT-4o mini, Mistral Small 3 24B, Phi-4 14B, Llama 3.1 8B, Qwen 2.5 7B, and Gemma 2 2B. Each model has 1,000 simulations over 50 scenarios. The user receives a fixed precomputed score for its trope, while the chatbot answers a 50-item IPIP Big Five inventory before and after context. The paper correlates the user's initial score with the chatbot's claimed change and finds the most consistent positive associations for Agreeableness and Conscientiousness; Emotional Stability is weakest. It also examines explicit amplify and resist prompts, longer conversations, model size, multivariable regression, persona recognition, and a WildChat-based extension. The central descriptive evidence is genuine and relevant: when the released outputs are aggregated over the 99 unique personas, many same-trait correlations remain or become larger. The implementation nevertheless does not demonstrate an internal or persistent personality change. Every dialogue turn and questionnaire item is a fresh API call; the only continuity is the complete transcript pasted into the prompt. The chatbot pre-test is also run only once per model and the same value is subtracted from all 1,000 observations. Subtracting a constant leaves correlation unchanged, so the primary result relates repeated user profiles to transcript-conditioned post-test self-assessments. There is no memory, persistent session, weight update, transfer without the transcript, or external judge. The 1,000 rows reuse 99 user profiles and 50 scenarios, yet tests treat them as independent without persona/scenario clustering or correction across 25 correlations per model. No neutral-transcript, shuffled-speaker, user-only, chatbot-only, or blinded-rater control separates interpersonal adaptation from reading, imitation, or lexical priming. The repository publishes 12,301 text outputs and supports meaningful checking, but base outputs are incomplete for Phi-4 and Mistral; the regression table does not reproduce from public CSVs; evolution and recognition outputs are absent; and the WildChat extension releases only 300 of 600 cases without the filtered dataset, while a write-mode bug erases transcripts from output files. That extension also asks GPT-4o mini to read a conversation retrospectively rather than measuring change in the original chatbot. The faithful conclusion is that dialogue context containing synthetic user personas robustly influences subsequent psychometric responses in several LLMs. This matters for consistency, alignment, and user experience, but it does not establish that models acquire a stable personality.

Español

Este trabajo estudia si la persona de un usuario modifica cómo un chatbot expresa y declara rasgos Big Five después de una conversación. GPT-4o mini representa uno de 100 tropos extremos nominales, en realidad 99 únicos porque uno está duplicado, y conversa durante 20 turnos con siete chatbots: GPT-4o, GPT-4o mini, Mistral Small 3 24B, Phi-4 14B, Llama 3.1 8B, Qwen 2.5 7B y Gemma 2 2B. Hay 1.000 simulaciones por modelo sobre 50 escenarios. El usuario recibe una puntuación fija precomputada por tropo y el chatbot responde antes y después al IPIP Big Five de 50 ítems. El paper correlaciona la puntuación inicial del usuario con el supuesto cambio del chatbot y encuentra asociaciones positivas especialmente consistentes en Agreeableness y Conscientiousness; Emotional Stability es la dimensión más débil. También prueba órdenes de amplificar o resistir el mimetismo, conversaciones más largas, tamaños de modelo, regresión multivariable, reconocimiento de persona y una extensión basada en WildChat. La evidencia descriptiva central es real y relevante: al reagrupar los outputs públicos por las 99 personas únicas, muchas correlaciones diagonales se mantienen o aumentan. Sin embargo, la implementación no demuestra un cambio interno o persistente de personalidad. Cada turno y cada ítem son llamadas API nuevas; la única continuidad es el transcript completo pegado al prompt. Además, el pre-test del chatbot se ejecuta una sola vez por modelo y ese mismo valor se resta a las 1.000 observaciones, de modo que restar la constante no cambia la correlación: el resultado principal relaciona el perfil repetido del usuario con la autoevaluación posterior condicionada por texto. No hay memoria, sesión persistente, actualización de pesos, transferencia sin transcript ni jueces externos. Las 1.000 filas reutilizan 99 perfiles y 50 escenarios, pero los tests las tratan como independientes, sin corrección por clúster ni por 25 comparaciones por modelo. Tampoco hay controles con transcript neutral, etiquetas intercambiadas, solo usuario, solo chatbot o evaluador ciego que separen imitación, lectura y priming léxico. El repositorio publica 12.301 outputs de texto y permite auditar mucho del patrón, pero los datos base están incompletos para Phi-4 y Mistral; la tabla de regresión no se reproduce desde los CSV públicos; faltan los outputs de evolución y reconocimiento; y WildChat publica solo 300 de 600 casos, sin el dataset filtrado, mientras un bug borra el transcript de los archivos. Además, esa extensión pide a GPT-4o mini leer retrospectivamente una conversación, no mide el cambio del chatbot original. La conclusión fiel es que el contexto conversacional con personas sintéticas influye de forma robusta en respuestas psicométricas posteriores de varios LLM. Esto importa para consistencia, alineamiento y experiencia de usuario, pero no prueba que los modelos adquieran una personalidad estable.

Research question

To what extent does the Big Five personality of a synthetic user influence the subsequent psychometric responses of different chatbots after a prolonged conversation, which traits and models are most sensitive, and can explicit instructions, duration, scale, or real conversations modulate or confirm this effect?

Method

GPT-4o mini represents extreme personality tropes and converses with each chatbot in one of 50 scenarios for ten turns per interlocutor. For each of seven models, 1,000 conversations are nominally generated. The user is associated with a precalculated Big Five vector per trope; the chatbot receives a single baseline per model and then answers, in 50 independent calls, the IPIP Big Five after reading the transcript. 25 Pearson correlations are calculated between user traits and post-pre differences of the chatbot, with Fisher intervals and p<0.05. The extensions test persona recognition, linear regression, amplification and resistance prompts, 60 turns, size variants, and WildChat transcripts. The audit read and visually reviewed the 19 pages, examined commit 3b885192, compiled the 22 scripts, inventoried 12,301 outputs and 12 CSV files, recalculated correlations of main results, interventions, sizes, and WildChat, reaggregated by persona, and reproduced the regression pipeline over the available artifacts.

Sample: The main design declares 1,000 conversations for each of seven chatbots, with 20 turns per conversation and 50 post-test items. These rows are not independent participants: they reuse 99 unique personas (the nominal list of 100 contains an exact duplicate) and 50 scenarios, and each persona retains a fixed Big Five vector. The public outputs contain 1,000 valid cases for five models, 949 valid out of 950 for Phi-4, and 551 for Mistral. Amplify and resist use 500 conversations; evolution uses 100 dialogues of 60 turns; the paper declares 600 WildChat cases but only publishes a folder of 300.

Findings

  • The published same-trait correlations are positive in most models; Agreeableness averages approximately 0.379 and Conscientiousness 0.345, while Emotional Stability is the weakest dimension, around 0.148.
  • The user's Emotional Stability trait is negatively associated with the chatbot's subsequent Agreeableness, with an approximate mean of -0.302 across models.
  • The descriptive signal does not disappear when averaging each unique persona: for example, Agreeableness reaches approximately 0.685 in Qwen 7B, 0.601 in Gemma 2B, and 0.632 in Mistral; Conscientiousness reaches 0.754, 0.585, and 0.508.
  • The amplify prompt increases several correlations (public diagonal approximately 0.620/0.380/0.549/0.371/0.389), confirming high sensitivity to an explicit instruction to imitate.
  • The resist prompt maintains lower but positive correlations in four of five dimensions; this shows competition between instructions and context, not by itself a deeply internalized adaptation.
  • The paper reports that most of the change measured in Mistral occurs during the first turns, but does not publish the raw evolution outputs to reproduce it.
  • Scale comparisons are not monotonic: some large variants correlate more, but quantized Llama 70B presents diagonals near zero or negative in the published artifact.
  • Persona verification reports 90.6% against random distractors and 77.6% against similar distractors; the judge is the same GPT-4o mini that generates the user and there is no human validation.
  • Five base folders publish 1,000 valid results, but Phi-4 only 949 and Mistral 551; the aggregated CSV retains complete rows not reconstructable from those outputs.
  • The public regression CSVs do not reproduce Table 3: the pair of 5,925 rows produces R2 approximately 0.692/0.356/0.437/0.576/0.606, versus 0.25/0.60/0.42/0.64/0.46 published.
  • The published WildChat folder of 300 cases produces approximate diagonal 0.367/0.173/0.383/0.090/0.367, but another 300 outputs and the filtered input are missing, so n=600 cannot be verified.
  • The 22 Python scripts compile, but there is no dependency environment, tests, CI, repository license, or integral reproducible command.

Limitations

  • Each turn and each item are new calls conditioned by the transcript; there is no persistent state, memory, or model update.
  • The chatbot pre-test is run once per model and reused as a constant for all conversations; the correlations are equivalent to correlating user with post-test.
  • The 1,000 rows repeat 99 user profiles and 50 scenarios, but the inference does not model clusters or crossed factors.
  • 25 pairs of traits are tested per model with p<0.05 without correction for multiple comparisons.
  • The user's initial scores come from an average per trope; the response files that originate this average are not published.
  • The post-test asks to consider the conversation and includes the full text, which allows priming, reading comprehension, and direct imitation.
  • There are no controls with neutral or random transcript, swapped labels, user-only or chatbot-only context, or blind external evaluation.
  • The same GPT-4o mini generates users and recognizes personas; there is no test with human readers and the recognition outputs are missing.
  • The regression fits StandardScaler before the split and splits rows at random, filtering the same personas, scenarios, and models between train and test.
  • The public regression artifacts do not correspond to the seven models and 7,000 rows described in the paper.
  • The main outputs are incomplete for Phi-4 and Mistral and there is no raw data for the temporal trajectory.
  • WildChat measures GPT-4o mini reading a transcript retrospectively, not the original chatbot; half of the outputs and the filtered dataset are missing.
  • A bug reopens each WildChat file in write mode and deletes the transcript, leaving only scores.
  • A trailing comma converts the chatbot's persona into a single-element tuple in the actual prompt, although the log shows a clean string; another obsolete index mislabels the footer.
  • Unrecognized responses are silently converted to the neutral value 3 and are not recorded per item.
  • There are no seeds, exact OpenAI snapshots, or fixed balance; temperature 0.7 and mutable aliases are used.
  • The repository lacks license, requirements, lockfile, tests, and CI; it contains local paths, private endpoint, placeholders, and unsafe eval.

What the study does not establish

  • It does not demonstrate an internal, persistent, or autonomous personality change of the LLM.
  • It does not demonstrate that the effect survives removing the transcript or transfers to an unconditioned task.
  • It does not administer an independent pre-test of the chatbot in each conversation.
  • It does not convert 1,000 repeated rows into 1,000 independent personas or justify the published nominal precision.
  • It does not have 100 unique personas: one description is duplicated and 99 distinct strings remain.
  • It does not validate the model's self-evaluation as personality perceived by external observers.
  • It does not causally separate interpersonal adaptation from lexical priming, transcript reading, or generic obedience.
  • It does not reproduce the regression table from the public CSVs.
  • It does not validate a real chatbot personality change in WildChat nor allow reproducing the 600 declared cases.
  • It does not demonstrate that a larger size always produces greater adaptation.
  • It does not test that the failure of resist implies a deeply internalized mechanism.
  • It does not establish generalization to human users, other languages, long deployments, or stable snapshots.

Traceability

Scope: Full text

Version: EMNLP 2025 main proceedings, pages 17314-17332, DOI 10.18653/v1/2025.emnlp-main.875; 19-page paper and official repository commit 3b885192 audited

Consulted source: https://aclanthology.org/2025.emnlp-main.875/

Review: Codex complete bilingual fidelity pass using the full EMNLP paper, all-19-page visual inspection, official repository commit audit, complete released-output inventory, independent main/intervention/size/WildChat correlation checks, persona-level reaggregation, executable regression reproduction, code-path and construct review; summaries written from full evidence rather than abstract keywords, 2026-07-15

Approval: Codex fidelity pass, 2026-07-15

English translation: approved, 2026-07-18

Models evaluated

  • GPT-4o
  • GPT-4o mini
  • Mistral Small 3 24B Instruct 2501
  • Phi-4 14B
  • Llama 3.1 8B Instruct
  • Qwen 2.5 7B Instruct
  • Gemma 2 2B Instruct
  • Phi-4 mini
  • Gemma 2 9B Instruct
  • Llama 3.1 70B Instruct quantized
  • Qwen 2.5 14B Instruct

Instruments and metrics

  • 50-item IPIP Big Five markers
  • Five-point Likert scoring
  • Pearson correlations
  • Fisher-z confidence intervals
  • Persona-recognition multiple choice
  • Linear regression
  • Root mean squared error
  • Coefficient of determination

Data used

  • Controlled synthetic conversations
  • 99 unique extreme personality trope strings
  • 50 fixed conversational scenarios
  • WildChat transcripts
  • 12,301 released text outputs
  • Official xingdom/chameleon-llms repository at commit 3b885192

Evidence and location

  • Question, method, main results, and declared limits: Official PDF pages 1-9, Abstract, Introduction, Experimental Setup, Results and Discussion
  • Models, 100 nominal personas, 50 scenarios, 1,000 conversations, and 20 turns: Official PDF pages 3-4 and 13-15, Sections 3.1-3.4 and Appendix A-B
  • IPIP, correlations, intervals, and results per trait: Official PDF pages 4-7 and 15-18, Sections 3.3-4.4, Figures 2-6 and appendix tables
  • Persona recognition, amplify, resist, evolution, and size: Official PDF pages 7-9 and 15-19, Sections 4.5-4.7 and Appendices C-F
  • WildChat and limits on humans and external evaluation: Official PDF pages 9-12 and 18-19, Section 4.8, Limitations and Appendix G
  • Calls without persistent state, single baseline, precalculated profiles, and logging bugs: Official repository commit 3b885192, base_experiment.py, base_personality.py and conversation helpers
  • 99 unique personas, repetition, and reanalysis aggregated by persona: Official repository trope lists, averaged_results.json and independently recalculated released base outputs
  • Output counts, main results, interventions, and scale: Complete audit of 12,301 released .txt outputs and 12 CSV files at commit 3b885192
  • Regression discrepancy and split leakage: linear_reg.py, features.csv, outputs.csv, new_features CSVs and independent regression execution
  • Incomplete WildChat, overwrite, and retrospective construct: base_wildchat.py, base_analysis_wildchat.py and released 300-file wildchat_outputs folder
  • Integral audit of context, validity, and artifact: reports/verification/article-198-context-conditioning-and-artifact-audit.json
  • Complete visual inspection: All 19 official PDF pages rendered and visually inspected on 15 July 2026