LLM Agents in Interaction: Measuring Personality Consistency and Linguistic Alignment in Interacting Populations of Large Language Models

Society, culture, and collective behavior2024ACL AnthologyApproved editorial review

Authors: Ivar Frisch, Mario Giulianelli

Keywords: Large Language Models, Agent interaction, Personality consistency, Linguistic alignment, Dialogue-based interaction

Source: Open primary source (opens in a new tab)

2
Authors
16
Findings
68
Limitations
16
Evidence

Editorial summary

English

The paper explores whether agents obtained as stochastic samples from GPT-3.5-turbo-0613 retain a persona through a writing task and whether their language moves toward that of another agent. Two extreme prompts are constructed. The profile called creative is extraverted, agreeable, conscientious, neurotic, and open; the profile called analytical is introverted, antagonistic, unconscientious, emotionally stable, and closed. All five traits move together, and creative/analytical are operational labels rather than validated human profiles. Every temperature-.7 response is treated as a different agent. Before and after writing a personal story, the same instance answers all 44 Big Five Inventory items; 500–900-word stories are converted to counts from 62 LIWC 2007 categories. The interactive condition is a single one-way exchange: one agent writes first and the second receives that entire story inside its prompt before producing another.

Without interaction, profiles are strongly separated on four BFI traits, with more overlap on neuroticism. The creative group retains almost identical post-writing means: 35 extraversion, 41 agreeableness, 37 conscientiousness, 16 neuroticism, and 47 openness. The analytical group shifts after the story from 15/11/18/13/15 to 17/21/32/15/29. A logistic regression over LIWC counts reaches 98.5% ten-fold cross-validated accuracy in distinguishing the labels. Prominent associations include more positive emotion and inclusion for creative and more discrepancy, negative emotion, and insight for analytical. Because all five prompted traits are changed together and are nearly perfectly collinear with group, however, no individual language pattern can be attributed to one trait.

After collaborative writing, classifier accuracy falls to 66.15% and BFI–LIWC correlations move closer to zero. The paper interprets this convergence as linguistic alignment and argues that creative adapts more toward analytical. Creative BFI responses remain stable, whereas analytical responses lie between the initial values and the individual-writing condition: agreeableness 18, conscientiousness 26, neuroticism 17, and openness 22. The authors interpret this latter pattern as analytical inconsistency rather than explicit partner alignment. That distinction is reasonable within their results, but the design does not measure alignment within each pair or separate adaptation from copying the supplied text, conversational memory, or topic change.

The audit of the official repository, frozen at commit ca6e117eb5a904fa97115f8845ef8b74aa461b8a, finds 65 outputs per label and condition. The data strengthen several cautions that the paper only describes qualitatively. Although the prompt forbids mentioning traits, all 130 control stories contain at least one explicit persona term under a targeted check; the same occurs in 130/130 ANACREA, 124/130 CREAANA, and 124/130 CREACREA stories. The 98.5% result may therefore detect words such as extrovert, introvert, antagonistic, or conscientious rather than implicit personality expression. Moreover, 37 of 65 ANACREA pairs, 15 of 65 CREAANA pairs, and 47 of 65 CREACREA pairs contain exactly identical stories across agents. The accuracy drop may partly reflect literal reproduction of the story inserted into the prompt. LIWC features are raw counts rather than length-normalized rates, and cross-validation does not group interacting pairs, allowing related observations to be split across training and test folds.

The implementation keeps the first BFI, story, and second BFI in each instance's conversation memory. Post-task stability or change can thus reflect contextual retention of earlier answers rather than personality persistence outside the history. Before/after analyses use f_oneway and pooled Cohen's d as if samples were independent, although they are paired measurements of the same agent; the extensive LIWC and correlation exploration is not multiplicity-corrected. The paper declares gpt-3.5-turbo-0613, while code calls the mutable gpt-3.5-turbo alias; the declared snapshot was removed from the API on September 13, 2024, so an exact rerun is no longer possible. The repository provides code, data, and an MIT license, but no lockfile, seeds, retry history, or configuration that automatically regenerates n=65. The defensible contribution is exploratory evidence that two extreme Big Five prompts yield different BFI and vocabulary distributions and that exposing a second model to a complete story reduces that separability. It does not establish human-like personality, psychometric consistency, bidirectional conversational adaptation, or a valid simulation of human populations.

Español

El trabajo explora si agentes obtenidos como muestras estocásticas de GPT-3.5-turbo-0613 mantienen una persona durante una tarea de escritura y si su lenguaje se aproxima al de otro agente. Los autores crean dos prompts extremos. La persona denominada creative es extrovertida, agradable, responsable, neurótica y abierta; la denominada analytical es introvertida, antagonista, poco responsable, emocionalmente estable y cerrada. Los cinco rasgos cambian a la vez y los nombres creative/analytical son etiquetas operativas, no perfiles humanos validados. Cada salida a temperatura 0,7 se trata como un agente distinto. Antes y después de escribir una historia personal, la misma instancia responde los 44 ítems del Big Five Inventory; las historias de 500 a 900 palabras se convierten en recuentos de 62 categorías LIWC 2007. La condición interactiva es un único intercambio unidireccional: un agente escribe primero y el segundo recibe íntegramente esa historia dentro de su prompt para producir otra.

En la condición sin interacción, los perfiles quedan muy separados en cuatro rasgos BFI; neuroticismo se solapa más. El grupo creative conserva prácticamente sus medias después de escribir: 35 en extraversión, 41 en amabilidad, 37 en responsabilidad, 16 en neuroticismo y 47 en apertura. El grupo analytical se desplaza después de la historia desde 15/11/18/13/15 hasta 17/21/32/15/29. Con recuentos LIWC, una regresión logística con validación cruzada de diez pliegues alcanza 98,5 % de exactitud para distinguir las dos etiquetas. Las asociaciones principales presentan más emoción positiva e inclusión en creative y más discrepancia, emoción negativa e insight en analytical. Sin embargo, estos cinco rasgos están manipulados en bloque y casi perfectamente correlacionados con el grupo, por lo que no puede atribuirse cada patrón a un rasgo individual.

Después de la escritura colaborativa, la exactitud del clasificador baja a 66,15 % y las correlaciones entre BFI y LIWC se acercan a cero. El artículo interpreta esa convergencia como alineación lingüística y afirma que creative se adapta más a analytical. Las respuestas BFI de creative siguen estables; las de analytical quedan entre el punto inicial y la condición de escritura individual: amabilidad 18, responsabilidad 26, neuroticismo 17 y apertura 22. Los autores interpretan este último patrón como inconsistencia de analytical, no como alineación explícita con la pareja. Es una distinción razonable dentro de sus resultados, pero el diseño no mide alineación dentro de cada pareja ni permite separar adaptación, copia del texto recibido, memoria conversacional o simple cambio de tema.

La auditoría del repositorio oficial, congelado en el commit ca6e117eb5a904fa97115f8845ef8b74aa461b8a, encuentra 65 salidas por etiqueta y condición. Los datos fortalecen varias cautelas que el PDF solo menciona de forma cualitativa. Aunque el prompt ordena no mencionar rasgos, los 130 relatos de control contienen al menos un término explícito de la persona según una comprobación dirigida; ocurre en 130/130 historias de ANACREA, 124/130 de CREAANA y 124/130 de CREACREA. Por ello, el 98,5 % puede reflejar palabras como extrovert, introvert, antagonistic o conscientious, no una expresión implícita de personalidad. Además, 37 de 65 parejas ANACREA, 15 de 65 CREAANA y 47 de 65 CREACREA contienen historias exactamente idénticas entre los dos agentes. La caída del clasificador puede proceder en parte de reproducción literal del relato insertado en el prompt. Los recuentos LIWC no se normalizan por longitud y la validación cruzada no agrupa parejas, de modo que observaciones relacionadas pueden repartirse entre entrenamiento y prueba.

La implementación mantiene en memoria el primer BFI, la historia y el segundo BFI de cada instancia. Así, estabilidad o cambio posterior puede ser retención contextual de respuestas previas, no persistencia de una personalidad fuera del historial. Los análisis antes/después usan f_oneway y d de Cohen para muestras independientes, aunque son mediciones emparejadas del mismo agente; no se corrige la extensa exploración de categorías LIWC y correlaciones. El PDF declara el snapshot gpt-3.5-turbo-0613, pero el código llama al alias mutable gpt-3.5-turbo; el snapshot declarado fue retirado del API el 13 de septiembre de 2024, por lo que hoy no puede repetirse literalmente. El repositorio aporta código, datos y licencia MIT, pero no lockfile, semillas, historial de reintentos ni una configuración que regenere automáticamente n=65. La contribución defendible es evidencia exploratoria de que dos prompts Big Five extremos producen distribuciones de BFI y vocabulario distintas, y de que exponer un segundo modelo a una historia completa reduce esa separabilidad. No prueba personalidad humana, consistencia psicométrica, adaptación conversacional bidireccional ni simulación válida de poblaciones humanas.

Research question

Can a single GPT-3.5 sampled as a population consistently express two extreme Big Five profiles in questionnaires and narratives, and does that expression change or align linguistically when one agent receives the narrative of another?

Method

Two experiments with GPT-3.5 at temperature 0.7. Each sample is considered an agent and receives one of two prompts that simultaneously manipulate the five Big Five traits. In the control, the same instance completes BFI-44, writes a personal story of 500–900 words, and repeats BFI. In the interactive condition, one agent writes and the second receives that complete narrative before writing their own; afterwards both repeat BFI. BFI sums, ANOVA and Cohen's d, counts of 62 LIWC 2007 categories, PCA, logistic regression with ten-fold cross-validation, point-biserial and Spearman correlations are analyzed. The audit read and rendered the ten pages, verified tables and appendices, and examined the code and official data at the commit level.

Sample: The article does not directly declare n in the method. The official data contain 65 outputs per label in each of four files of 130 rows: control, analytical→creative, creative→analytical and creative→creative, for 520 stored stories. The central results use the control and crossed conditions; the current configuration of the scripts only generates 10 analytical subjects in control and one per profile in each interactive condition, so n=65 requires unrecorded manual changes. Each row is a stochastic sample of the same model, not a person or independent trained model.

Findings

  • The extreme prompts produce a very large separation between labels in four of five BFI sums before writing.
  • Neuroticism shows the smallest initial separation and more overlapping distributions.
  • The creative profile keeps the five BFI means practically the same after individual writing.
  • The analytical profile increases its five BFI means after individual writing.
  • In analytical, agreeableness goes from 11 to 21, conscientiousness from 18 to 32 and openness from 15 to 29 after writing.
  • Logistic regression on LIWC counts obtains 98.5% mean accuracy without interaction.
  • Creative uses more positive emotion and inclusion categories in the individual condition.
  • Analytical uses relatively more discrepancy, negative emotion and insight in the individual condition.
  • After the exchange of one story, the classifier accuracy drops to 66.15%.
  • The published BFI–LIWC correlations are generally weaker after the interactive condition.
  • Creative retains its BFI means also after the interactive condition.
  • Analytical remains after interaction between its initial point and its large individual writing shift in four traits.
  • The authors interpret the BFI change of analytical as inconsistency, not as conclusive alignment with creative.
  • The official data contain explicit mentions of traits in the 130 control stories despite the prompt prohibition.
  • Between 15 and 47 of 65 pairs per interactive condition store exactly identical stories between agents.
  • The official repository provides code, processed data, LIWC dictionary, figures and MIT license.

Limitations

  • A single closed model and a specific 2023 snapshot are evaluated.
  • Each stochastic output is called an agent without demonstrating persistent identity or independence equivalent to a human population.
  • The claim of human-like variability is not contrasted with a human sample.
  • The creative and analytical profiles simultaneously manipulate the five Big Five traits.
  • The five traits are almost perfectly confounded with the binary group label.
  • It is not possible to attribute a LIWC category or a behavioral change to an individual trait.
  • The creative and analytical labels are not validated as creativity, analytical thinking or real human profiles.
  • Both profiles are extreme and the article itself recognizes that they do not represent common human combinations.
  • Intermediate levels, mixed profiles or single-trait manipulation are not tested.
  • The BFI is administered via self-report to a model that has just explicitly received the expected traits.
  • The correct BFI answer can be deduced directly from the prompt without expressing a latent construct.
  • Internal consistency, test-retest reliability, factor structure or convergent/discriminant validity are not evaluated.
  • The appendix claims minimum 0 and maximum 50 for each trait, but the implemented sum has minimums 8–10 and maximums 40–50 depending on the number of items.
  • The scores present very strong ceilings and floors, which questions parametric assumptions.
  • The same conversational memory contains the first BFI, the story and the second BFI.
  • The second BFI may repeat previous answers from memory rather than measuring independent stability.
  • Changes after writing may be context, recency or story topic effects.
  • Collaborative writing consists of a single unidirectional dialogue turn.
  • Only the second agent receives content from the partner; the first cannot align with a future response.
  • The partner's complete narrative is inserted literally into the second agent's prompt.
  • There is no control condition that receives an unrelated or shuffled third-party story.
  • Copy, lexical priming, adaptation, memory and thematic change are not separated.
  • Alignment is not calculated as change within each pair relative to its own baseline.
  • The accuracy drop of a group classifier is not by itself a causal metric of conversational alignment.
  • The article does not measure sequential, turn-by-turn, syntactic or semantic alignment.
  • Between 23% and 72% of the pairs, depending on condition, contain exactly identical stories between the two agents.
  • Exact copies are neither excluded nor analyzed separately.
  • High copying may artificially reduce separability between labels after interaction.
  • The prompt requests 800 words, but accepts any story between 500 and 900 and the observed means range around 628–656.
  • The data contain explicit mentions of person terms in almost all stories.
  • In control, the 130 stories contain at least one explicit term under the audit's directed check.
  • The 98.5% classifier may exploit direct trait names rather than implicit style.
  • LIWC is calculated as raw counts, not frequencies normalized by length.
  • Length differences may influence PCA, correlations and classification.
  • The regression standardizes the entire set before cross-validation, introducing preprocessing information leakage.
  • Cross-validation does not cluster the two observations of an interactive pair.
  • Related or identical stories may be split between training and test.
  • Deviation across folds, intervals, confusion matrix or permutation comparison are not reported in the article.
  • The two-dimensional PCA is descriptive and does not validate statistical separation or convergence.
  • Before/after analyses use independent-samples ANOVA although the measurements are paired by agent.
  • The implemented Cohen's d uses independent pooled deviation instead of an effect for paired data.
  • The dependence of agents within interactive pairs is not modeled.
  • Normality, homoscedasticity, robustness or sensitivity to the bounded nature of BFI sums are not checked.
  • Dozens of LIWC categories are explored against groups and five traits without integral correction for multiple comparisons.
  • The correlation tables select the top five coefficients without publishing the full family in the PDF.
  • p-values do not accompany the selected correlations in tables 4 and 7.
  • The correlation between BFI and LIWC may reflect the binary separation of prompts that determines both results.
  • LIWC 2007 is not validated for generated stories that explicitly name personality instructions.
  • Human judges of personality, narrative quality, naturalness or alignment are not included.
  • Neutral prompts, descriptions without Big Five, human scripts or simple lexical rules are not compared.
  • There is no analytical→analytical condition; the available same-person control is only creative→creative.
  • The CREACREA condition present in the data is not described or analyzed transparently in the main article.
  • The article does not declare n directly nor offers power analysis or preregistration.
  • The published code uses the alias gpt-3.5-turbo, not the snapshot gpt-3.5-turbo-0613 declared in the PDF.
  • The mutable alias may have pointed to different versions depending on the execution date.
  • The declared snapshot was withdrawn from the API and no longer allows literal repetition.
  • The current scripts fix subject_count at 10 or 1 and require manual editing to regenerate the 65 outputs per group.
  • There is no installable requirements file, lockfile, container or automated environment; versions only appear in README.
  • Seeds, exact execution dates, regions, top-p or other resolved values from provider defaults are not published.
  • Retries for format and length are run until success without recording failure rates or discarded candidates.
  • Filtering by length and format may introduce unquantified selection.
  • The README recommends manually deleting rows after errors, which weakens generation traceability.
  • The code requests inserting an API key into files and prints it, an unsafe practice for reproducibility.
  • Paths and scripts require manual changes to alternate analytical conditions.
  • The processed results from the repository are not linked to a single immutable pipeline that regenerates each table.
  • Cost, latency or total number of calls and retries are not reported.
  • Other languages, tasks, lengths, domains, models or human-AI interactions are not evaluated.
  • The ethics section identifies malicious personalization and toxic content, but does not implement technical safeguards.

What the study does not establish

  • It does not demonstrate that GPT-3.5 possesses personality.
  • It does not demonstrate that each stochastic sample is an independent agent comparable to a person.
  • It does not validate the creative and analytical profiles as human personality types.
  • It does not identify the causal effect of each Big Five trait separately.
  • It does not establish reliability or psychometric validity of BFI responses.
  • It does not demonstrate implicit personality when the narratives explicitly name the traits.
  • It does not separate personality stability from conversational memory and answer repetition.
  • It does not separate linguistic alignment from literal copy of the partner's text.
  • It does not demonstrate reciprocal interaction or multi-turn dialogue.
  • It does not test that creative causally adapts more than analytical under a complete symmetric design.
  • It does not demonstrate that lower classifier accuracy represents more human behavior.
  • It does not establish generalization to other models, profiles, languages or tasks.
  • It does not validate simulation of human populations or psychological conclusions about people.
  • It does not evaluate efficacy, safety or benefit in a real application.
  • It does not offer today an exact reproduction of the experiment with the original snapshot available.

Traceability

Scope: Full text

Version: ACL Anthology 2024.personalize-1.9; PERSONALIZE 2024; DOI 10.18653/v1/2024.personalize-1.9; CC BY 4.0

Consulted source: https://aclanthology.org/2024.personalize-1.9.pdf

Review: Codex full-text, visual, psychometric, statistical, code, data-integrity and reproducibility audit, 2026-07-15

Approval: Codex fidelity pass, 2026-07-15

English translation: approved, 2026-07-18

Models evaluated

  • GPT-3.5-turbo-0613 as declared in the paper
  • gpt-3.5-turbo mutable alias as used by the released code
  • Creative extreme Big Five prompted profile
  • Analytical extreme Big Five prompted profile
  • Stochastic outputs at temperature 0.7 treated as separate agents

Instruments and metrics

  • 44-item Big Five Inventory with 1–5 Likert responses
  • Creative and analytical five-trait persona prompts
  • Individual 800-word personal-story prompt with 500–900-word acceptance filter
  • One-way collaborative story prompt containing the partner's full response
  • LIWC 2007 English dictionary with 62 category counts
  • One-way ANOVA via scipy.stats.f_oneway
  • Pooled Cohen's d
  • Point-biserial and Spearman correlations
  • PCA visualization
  • Logistic regression with ten-fold cross-validation

Data used

  • Official 0_CONTROL dataset: 65 analytical and 65 creative outputs
  • Official 1_ANACREA dataset: 65 analytical-first and 65 creative-second outputs
  • Official 2_CREAANA dataset: 65 creative-first and 65 analytical-second outputs
  • Official 3_CREACREA dataset: two groups of 65 creative-profile outputs
  • LIWC count matrices, encoded prompts, BFI scores and published correlation outputs
  • Official Interaction_LLMs repository at ca6e117eb5a904fa97115f8845ef8b74aa461b8a

Evidence and location

  • Questions, scope and summary of results: PERSONALIZE 2024 paper, sections 1–2, pp. 102–103
  • Model, temperature, snapshot and population by sampling: PERSONALIZE 2024 paper, section 2.1 and footnote 2, p. 103
  • Extreme creative and analytical prompts: PERSONALIZE 2024 paper, section 2.2, p. 104 and Appendix A.1–A.2, p. 109
  • BFI-44 and scoring system: PERSONALIZE 2024 paper, section 2.3, p. 104 and Appendices A.4–A.6, pp. 109–110
  • Individual and interactive tasks and length filter: PERSONALIZE 2024 paper, section 2.4 and footnote 5, p. 104; Appendix A.3, p. 109
  • Initial BFI separation and shift after writing: PERSONALIZE 2024 paper, section 3.1.1, pp. 104–105; Tables 1–3, p. 110
  • LIWC accuracy of 98.5% and correlations without interaction: PERSONALIZE 2024 paper, section 3.1.2 and Figure 2, p. 105; Table 4, p. 111
  • BFI results after interaction: PERSONALIZE 2024 paper, section 3.2.1, p. 105; Figure 4 and Tables 5–6, pp. 110–111
  • Accuracy of 66.15%, LIWC convergence and interpreted asymmetry: PERSONALIZE 2024 paper, section 3.2.2 and Figure 3, p. 106; Table 7, p. 111
  • Recognized limitations and explicit mention issues: PERSONALIZE 2024 paper, Limitations, pp. 106–107
  • Risks and non-human profiles: PERSONALIZE 2024 paper, Ethical Considerations, pp. 107–108
  • Metadata, DOI, pages and license: ACL Anthology 2024.personalize-1.9 metadata; DOI 10.18653/v1/2024.personalize-1.9; CC BY 4.0
  • Code, model alias, memory, retries and statistical analysis: Official Interaction_LLMs repository, commit ca6e117eb5a904fa97115f8845ef8b74aa461b8a, generation and plot_stats scripts
  • n=65 per label, explicit mentions and exact copies between pairs: Official Interaction_LLMs repository data at commit ca6e117eb5a904fa97115f8845ef8b74aa461b8a; integrity audit 15 Jul 2026
  • Frozen snapshot of official repository: .cache/editorial-sources/article-073/supplements/audit/Interaction_LLMs-ca6e117eb5a904fa97115f8845ef8b74aa461b8a.tar.gz; sha256 81d2cb5a2328b5532c53532dd3861e2be5da20970736cd53b75a2651eb496817
  • Withdrawal of snapshot gpt-3.5-turbo-0613: OpenAI API deprecations: shutdown 13 Sep 2024; verified 15 Jul 2026