Synthetic Personality Review

Investigating the Personality Consistency in Quantized Role-Playing Dialogue Agents

Personas, identity, and agents2024ACL AnthologyApproved editorial review

Authors: Yixiao Wang, Homa Fashandi, Kevin Ferreira

Keywords: model quantization, role-playing agents, personality consistency, Big Five model, edge computing, multi-turn dialogue, conversational AI

Source: Open primary source (opens in a new tab)

3
Authors
15
Findings
35
Limitations
19
Evidence

Editorial summary

English

This paper studies whether four small LLMs retain prompt-assigned Big Five profiles across 20 narrative exchanges and whether quantization changes that stability. Each of the 32 binary trait combinations is paired with its opposite. LLaMA3 8B Instruct, Mistral 7B Instruct v0.3, Gemma 7B Instruct v1.1, and Gemma2 9B Instruct are run through Ollama in FP16, GGUF Q8_0, and Q4_0; every pair and condition is repeated 15 times. At each turn, agents collaborate on a story, repeat a 44-item BFI self-report, and produce narratives analyzed with LIWC and nomic-embed-text-v1 embeddings. Think2 merely adds to the narrative prompt: “Before writing the story, think twice what is your personality.” The plots show that opposite Gemma2 and LLaMA3 profiles tend to converge under the baseline and remain more separated with Think2; the authors recommend Gemma2+Think2 at Q4_0 and LLaMA3+Think2 at Q8_0. The effect is not uniform: Mistral loses much of its initial relationship under both methods, while Gemma 7B falls close to zero on the global metric at every precision. The claim that quantization invariably degrades personality is also not causally isolated: FP16 drifts too, LLaMA3 Q8_0 is relatively stable, and Think2 improves FP16 as well, supporting a generic persona reminder more directly than a quantization-specific correction. The metric called global correlation is not an ordinary Pearson coefficient: it concatenates pairs, sums the magnitudes of positive and negative correlations between five OCEAN dimensions and linguistic features, then applies min-max normalization. A value near 1 is relative to the compared set, not r=1. The paper publishes 24 figures but no numeric result tables, tests, intervals, or effect sizes; it also omits decoding details, BFI scoring, cross-validation splits, and a code or data release. This is exploratory evidence of drift in adherence to extreme prompts and of reminder-based reinforcement for some model families; it does not establish psychological personality, a general causal effect of quantization, or reliability with real users or edge devices.

Español

Este trabajo estudia si cuatro LLM pequeños mantienen perfiles Big Five impuestos por prompt durante 20 intercambios narrativos y si la cuantización modifica esa estabilidad. Cada una de las 32 combinaciones binarias de rasgos se empareja con su opuesta. LLaMA3 8B Instruct, Mistral 7B Instruct v0.3, Gemma 7B Instruct v1.1 y Gemma2 9B Instruct se ejecutan con Ollama en FP16, GGUF Q8_0 y Q4_0; cada pareja y condición se repite 15 veces. En cada turno los agentes colaboran en una historia, repiten un autoinforme BFI de 44 ítems y producen narraciones analizadas con LIWC y embeddings nomic-embed-text-v1. Think2 solo añade al prompt narrativo: “Before writing the story, think twice what is your personality”. Los gráficos muestran que los perfiles opuestos de Gemma2 y LLaMA3 tienden a acercarse con el baseline y que Think2 conserva mejor su separación; los autores recomiendan Gemma2+Think2 en Q4_0 y LLaMA3+Think2 en Q8_0. El efecto no es uniforme: Mistral pierde gran parte de la relación inicial con y sin Think2, y Gemma 7B cae casi a cero en la métrica global bajo todas las precisiones. La afirmación de que la cuantización degrada invariablemente la personalidad tampoco queda aislada causalmente: FP16 también deriva, Q8_0 de LLaMA3 es relativamente estable y Think2 mejora asimismo FP16, por lo que la evidencia respalda un recordatorio genérico de persona más que una corrección específica de cuantización. La métrica llamada global correlation no es un coeficiente Pearson ordinario: concatena parejas, suma magnitudes de correlaciones positivas y negativas entre cinco OCEAN y características lingüísticas y aplica normalización min-max. Un valor cercano a 1 es relativo al conjunto comparado, no r=1. El artículo publica 24 figuras pero ninguna tabla de resultados numéricos, pruebas, intervalos o tamaños de efecto; tampoco especifica decodificación, scoring BFI, particiones de validación cruzada ni libera código o datos. Es evidencia exploratoria de deriva de cumplimiento de prompts extremos y de que un recordatorio puede reforzarlo en algunas familias; no demuestra personalidad psicológica, impacto causal general de la cuantización ni fiabilidad en usuarios o dispositivos reales.

Research question

How do the Big Five profiles assigned to role-playing agents built with 7-9B LLMs in FP16, Q8_0, and Q4_0 change over 20 turns; can an in-context reminder called Think2 reduce that drift; and which combination of model, precision, and prompt best preserves the differences between opposite profiles?

Method

The authors encode each OCEAN dimension as a negative or positive adjective and generate the 32 binary combinations, organized into 16 opposite pairs. Each agent completes the BFI and narrates a story; afterwards, both members collaborate for 20 turns, see the last opposing response, and repeat the self-report and narration. Baseline and Think2, which adds a reflection sentence about personality, are compared. Four models by three precisions and 16 pairs are repeated 15 times. The analysis uses OCEAN score radars, linear regression with LIWC and embeddings to separate profiles through cross-validation, and a normalized global metric based on correlations between initial scores and narrative features.

Sample: The study is entirely synthetic. Its design crosses four 7-9B models, three numeric representations, two prompts, 32 extreme Big Five profiles grouped into 16 pairs, 21 measurement moments, and 15 repetitions. The agents interact only with the exactly opposite profile and write collaborative personal stories. There are no users, human annotators, natural characters, or peripheral devices evaluated; the article also does not identify the final number of usable narrations after possible output failures.

Findings

  • Profiles are induced through five explicit adjectives in the system prompt, one per Big Five dimension.
  • The 32 binary combinations cover all high/low patterns and are paired with their bitwise opposite.
  • The agent repeats the 44-item BFI in each of the 20 turns in addition to the initial measurement.
  • Think2 differs from the baseline by a single instruction to recall personality before writing.
  • In the Gemma2 examples, the scores of opposite profiles converge after 20 turns with baseline and remain more separated with Think2.
  • Embedding separability is usually high from turn 0 and is better maintained than LIWC.
  • Think2 raises the cross-validation accuracy of Gemma2 and LLaMA3 in many visualized conditions.
  • For Gemma2, the baseline global correlation drops in FP16, Q8_0, and Q4_0, while Think2 preserves relative levels around 0.5-0.7.
  • For LLaMA3, the Q8_0 baseline is more stable than FP16 and Q4_0, and Think2 improves especially FP16 and Q8_0.
  • Mistral loses much of the correlation after the initial turn across all precisions; Think2 provides small improvements.
  • Gemma 7B drops to nearly zero in the global correlation with both baseline and Think2, so the reminder does not rescue all families.
  • The authors designate Gemma2+Think2 Q4_0 as the best four-bit option and LLaMA3+Think2 Q8_0 as the best eight-bit option.
  • The improvement of Think2 also appears in FP16, indicating that it does not depend on quantization.
  • Results are communicated through 24 figures and qualitative description, without a numeric table allowing comparison across all conditions.
  • No official repository of code, data, or outputs is identified for recalculating the figures.

Limitations

  • The study measures maintenance of persona instructions, not the psychological personality of the model.
  • Each continuous Big Five dimension is reduced to two extreme adjectives and all gradation is lost.
  • Some poles are evaluative or conceptually imprecise, such as antagonistic/agreeable and unconscientious/conscientious.
  • The text calls 00000 'extremely analytical', although that code combines closedness to experience, low conscientiousness, introversion, antagonism, and emotional stability.
  • Always pairing with the exactly opposite profile maximizes interference and does not represent ordinary conversations.
  • There are no conditions with similar, neutral, human interlocutors, or continuous profiles.
  • FP16 also shows drift and Think2 also improves it, so the design does not separate conversational drift from the causal effect of quantization.
  • LLaMA3 Q8_0 may be more stable than FP16, contradicting a simple monotonic relationship between fewer bits and less consistency.
  • The claim of invariable degradation exceeds the visible heterogeneity across models and precisions.
  • Think2 is a generic prompt reminder; it is not compared with same-length controls, re-injecting the traits, asking to think twice without mentioning personality, or summarizing history.
  • The benefit may come from direct priming toward the labels that also define the target, not from stable reflection.
  • The BFI self-report is administered while the system prompt explicitly contains the traits, favoring responses by literal compliance.
  • The 44 items, the inversion key, the per-subscale calculation, and the treatment of malformed responses are not published.
  • The radars appear to use totals on a common axis up to 50 even though the BFI subscales have different numbers of items; normalization is not explained.
  • Repeating the same questionnaire 21 times within the context may reinforce or contaminate subsequent responses.
  • LIWC depends on a proprietary dictionary and no license, exact version of the resource, or categories used are reported.
  • High-dimensional embeddings may capture topic, style, model, or template in addition to traits.
  • The regression does not document the target variable, preprocessing, regularization, number of folds, grouping, or partitions.
  • If narrations from the same pair or repetition appear in training and test, cross-validation may have identity leakage; the article does not allow ruling this out.
  • The result of linear regression is called accuracy without formally defining how it is converted into a percentage.
  • The global metric concatenates all pairs and hides variation by trait, profile, and repetition.
  • Summing magnitudes of positive and negative correlations eliminates direction and may reward contradictory associations.
  • Min-max normalization makes the value depend on the compared set and not be comparable as Pearson r across figures or studies.
  • Intervals, hypothesis tests, p-values, multiplicity correction, and effect sizes are not detailed despite the repeated use of 'significant'.
  • Bands and boxplots are not accompanied by values, precise definition of error, or downloadable data.
  • Temperature, top-p, seed, exact story length, token limits, and failure policy are missing.
  • Exact file names and hashes of GGUF files, Ollama/ggml versions, hardware, and execution configuration are missing.
  • Real memory, latency, energy, response quality, or performance on a peripheral device are not measured.
  • The 'optimal' choice uses only these consistency metrics, not the full balance between resources, utility, safety, and quality.
  • Only four 7-9B models, one GGUF family, and two integer levels are studied, all in English.
  • There is no human evaluation of whether the narrations express the assigned trait or remain natural, useful, and coherent.
  • There are no users, demographics, realistic characters, diverse tasks, or interactions other than collaborative stories.
  • The privacy claim relies on theoretical local execution; storage, logs, telemetry, or threats are not audited.
  • There is no ethics section or analysis of risks of antagonistic personalities, imitation, manipulation, or emotional companions.
  • Without code, data, and numeric results, no model selection can be independently reproduced.

What the study does not establish

  • It does not demonstrate that LLMs possess internal Big Five traits.
  • It does not demonstrate that the BFI self-report is valid when the subject is an LLM instructed with the expected responses.
  • It does not demonstrate that quantization always or monotonically degrades consistency.
  • It does not isolate the effect of quantization from the effect of model, prompt, conversation, and sampling.
  • It does not demonstrate that Think2 is a specific mitigation for quantization.
  • It does not demonstrate that Think2 works in Gemma 7B or equally across all families.
  • It does not demonstrate statistical significance of the shown differences.
  • It does not allow interpreting the normalized global correlation as Pearson r.
  • It does not demonstrate that embedding separability equates to personality fidelity.
  • It does not demonstrate that consistent BFI responses predict behavior outside the narrative task.
  • It does not demonstrate stability under human conversations, long memory, tools, RAG, or real environments.
  • It does not demonstrate generalization to other models, bits, quantization formats, languages, or tasks.
  • It does not demonstrate that Gemma2 Q4_0 or LLaMA3 Q8_0 are optimal for edge in cost or global quality.
  • It does not demonstrate privacy improvements by themselves; it only presumes local inference.
  • It does not demonstrate safety, satisfaction, trust, or benefit for users.
  • It does not allow reconstructing the figures without experimental artifacts.
  • It does not establish that the conclusions hold with current versions of models and runtimes.

Traceability

Scope: Full text

Version: EMNLP 2024 Industry Track final proceedings paper, pp. 239-255, DOI 10.18653/v1/2024.emnlp-industry.19, 17 pages; no official code or data release identified in the paper, ACL record, or targeted project search

Consulted source: https://aclanthology.org/2024.emnlp-industry.19.pdf

Review: Codex full-text, bilingual-fidelity, visual, bibliographic, psychometric, quantization, metric-interpretation, experimental-design, reproducibility, privacy and ethics audit, 2026-07-15

Approval: Codex fidelity pass, 2026-07-15

English translation: approved, 2026-07-18

Models evaluated

  • LLaMA3 8B Instruct
  • Mistral 7B Instruct v0.3
  • Gemma 7B Instruct v1.1
  • Gemma2 9B Instruct
  • nomic-embed-text-v1 con Sentence Transformers, contexto 8192
  • Ollama como entorno de inferencia
  • GGUF FP16, Q8_0 y Q4_0 mediante ggml/llama.cpp

Instruments and metrics

  • Big Five Inventory (BFI), 44 ítems, autoadministrado al LLM en cada turno
  • Cinco dimensiones OCEAN codificadas como 32 perfiles binarios extremos
  • Linguistic Inquiry and Word Count 2015 (LIWC2015)
  • Embeddings nomic-embed-text-v1
  • Regresión lineal y exactitud de validación cruzada
  • Radares OCEAN con bandas de error
  • Métrica propia de correlación global con suma de magnitudes y normalización min-max

Data used

  • 32 perfiles OCEAN binarios definidos por prompt
  • 16 parejas de perfiles exactamente opuestos
  • 20 turnos de interacción más la medición inicial
  • 15 repeticiones por pareja, modelo, nivel de precisión y método
  • Narraciones personales y autoinformes generados por los propios LLM
  • No se usa ni publica un conjunto de datos humano

Evidence and location

  • Official bibliographic record: ACL Anthology 2024.emnlp-industry.19: 3 authors, EMNLP 2024 Industry Track, pp. 239-255, DOI 10.18653/v1/2024.emnlp-industry.19
  • Complete audited source: .cache/editorial-sources/article-096/source.pdf; official ACL PDF; 17 pages; sha256 0cab9a312f18849a502a4af2dcf492e8b2a99c6d788bd57f1273d22519485899
  • Research questions and claims: Full text pp. 239-240, Introduction
  • Four models and three precisions: Full text p. 241, section 3.1
  • Thirty-two binary profiles and sixteen pairs: Full text p. 241, section 3.2 and Algorithm 1
  • Story, interaction, and BFI prompts: Full text p. 242, Tables 1-2
  • Twenty turns and repeated self-evaluation: Full text p. 242, section 3.3
  • LIWC and embeddings: Full text pp. 242-243, section 3.4
  • Exact definition of Think2: Full text pp. 242-243, Narrative Task Prompt and section 3.5
  • Fifteen repetitions per pair: Full text p. 243, Experimental Results
  • OCEAN separation with and without Think2: Full text p. 243, Figure 2; Appendix pp. 250-251, Figures 5-11
  • Linguistic cross-validation: Full text p. 244, Figure 3 and section 4.2; Appendix pp. 251-255, Figures 12-23
  • Construction of global correlation: Full text p. 244, section 4.3 and Equation 1
  • Heterogeneity across families: Full text pp. 244-245, Figure 4; Appendix pp. 249 and 255, section C and Figure 24
  • Recommended choices: Full text p. 245, sections 4.4-5
  • Acknowledged limitations: Full text p. 246, section 6
  • Absence of numeric tables and tests: All results in full text and appendices are figures or qualitative prose; no numeric results table, uncertainty table, hypothesis test or effect size is reported
  • Absence of reproducible artifact: Paper and ACL metadata contain no project, code or data URL; targeted official GitHub/project search checked 15 Jul 2026
  • Comprehensive visual inspection: All 17 PDF pages rendered and visually inspected, including all 24 figures, two prompt tables, Algorithm 1, limitations and appendices; checked 15 Jul 2026