Synthetic Personality Review

Scaling Personality Control in LLMs with Big Five Scaler Prompts

Trait induction and control2026arXivApproved editorial review

Authors: Gunhee Cho, Yun-Gyung Cheong

Keywords: Computation and Language, Multiagent Systems, Big Five personality traits, Prompt engineering

Source: Open primary source (opens in a new tab)

2
Authors
7
Findings
14
Limitations
13
Evidence

Editorial summary

English

Cho and Cheong present Big5-Scaler, a family of instructions that encodes the five OCEAN traits as numerical x-out-of-n scores and asks a model to respond accordingly. Three variants are tested: simple, with one sentence per trait; specific, with six facet descriptions per trait; and simspec, combining both levels. The paper studies five different questions, not all with the same models or design. First, Alpaca-7B answers the 1,000-item MPI while one trait is maximized and is compared with a neutral condition and three prompting baselines. The simple Big5-Scaler prompt has the highest mean for openness, conscientiousness, extraversion, and agreeableness; for neuroticism, however, neutral (3.01) exceeds every induction and the best Big5-Scaler variant reaches 2.73. Second, LLaMA3-8B, a model labelled Mistral-25B, and Phi-4-14B answer the BFI, IPIP-NEO-120, and NEO-FFI while the target trait varies from 0 to 90 and the other four remain at 50. Many correlations between assigned and questionnaire scores are high, but there are clear failures, especially for LLaMA: openness under the IPIP simple prompt is r=-0.235, conscientiousness under the specific prompt is r=0.032, and BFI openness under the simple prompt is r=0.486. These tests assess whether a model reproduces explicit construct instructions in a self-report instrument that measures the same construct; they are not independent validation in open behavior. Third, the authors generate 20-turn dialogues between randomly profiled agents. Three LLM judges identify which of two agents expresses more of a trait with scores from 35.4% to 47.4%, against 33.3% expected when choosing A, B, or equal; the paper does not report the sample size, uncertainty, tests, or human evaluation for this task. Fourth, the concatenated first nine utterances of each agent are compared with the tenth: cosine similarity is 0.999 for all three models, BERTScore ranges from 0.480 to 0.537, and PersonaCLR from 0.789 to 0.828. Textual or stylistic similarity alone does not identify trait stability, and the near-perfect cosine lacks enough specification to interpret. Fifth, 17 Korean NLP graduate students complete IPIP-NEO-120; their scores become prompts and the model answers the same inventory. Published RMSE values of 1.785–1.822 only modestly improve on an approximate random value of 2.0, without uncertainty or a statistical comparison. An additional analysis creates 50 agents for each model, prompt type, and n in {10,25,50,100}; Phi-4-14B with the simple prompt and n=10 has the lowest average RMSE (21.587), only 0.020 below the same model at n=50. The narrative claim that n=10 wins consistently has an exception for Mistral-specific, whose best average is n=100, and several questionnaire-level exceptions. The paper provides useful evidence that explicit prompts control questionnaire answers and some linguistic signals, but it does not establish internal personality, human equivalence, or high-fidelity individual imitation.

Español

Cho y Cheong presentan Big5-Scaler, una familia de instrucciones que escribe los cinco rasgos OCEAN como puntuaciones numéricas de x sobre n y pide al modelo que responda de acuerdo con ellas. Hay tres variantes: simple, con una frase por rasgo; specific, con seis descripciones de facetas por rasgo; y simspec, que combina ambos niveles. El trabajo estudia cinco cuestiones distintas, pero no todas con los mismos modelos ni el mismo diseño. Primero, Alpaca-7B responde el MPI de 1.000 ítems cuando se maximiza un solo rasgo y se compara con una condición neutral y tres baselines de prompting. Big5-Scaler simple obtiene la mayor media en apertura, responsabilidad, extraversión y amabilidad; en neuroticismo, la condición neutral (3,01) supera a todas las inducciones y la mejor variante Big5-Scaler queda en 2,73. Segundo, LLaMA3-8B, un modelo denominado Mistral-25B y Phi-4-14B contestan BFI, IPIP-NEO-120 y NEO-FFI mientras el rasgo objetivo varía de 0 a 90 y los otros cuatro quedan en 50. Muchas correlaciones entre puntuación indicada y cuestionario son altas, pero hay fallos claros, sobre todo en LLaMA: por ejemplo, apertura con IPIP bajo prompt simple r=-0,235, responsabilidad con prompt specific r=0,032 y apertura con BFI simple r=0,486. Estas pruebas miden si el modelo reproduce en un autoinforme instrucciones que ya nombran y describen el mismo constructo; no son una validación independiente en conducta abierta. Tercero, se generan diálogos de 20 turnos entre agentes con perfiles aleatorios. Tres jueces LLM aciertan cuál de dos agentes expresa más un rasgo entre 35,4% y 47,4%, frente al 33,3% esperado al elegir entre A, B o igual; el artículo no aporta tamaño de muestra, intervalos, pruebas ni evaluación humana para esta tarea. Cuarto, se compara la concatenación de los nueve primeros turnos de cada agente con el décimo: el coseno es 0,999 para los tres modelos, BERTScore queda entre 0,480 y 0,537 y PersonaCLR entre 0,789 y 0,828. Esa semejanza textual o estilística no identifica por sí sola estabilidad de rasgo, y el coseno casi perfecto carece de especificación suficiente para interpretarlo. Quinto, 17 estudiantes de posgrado coreanos de NLP completan IPIP-NEO-120; sus scores se convierten en prompts y el modelo vuelve a contestar el mismo inventario. Los RMSE publicados, 1,785–1,822, mejoran solo modestamente un azar aproximado de 2,0, sin incertidumbre ni contraste estadístico. Un análisis adicional crea 50 agentes por combinación de modelo, tipo de prompt y escala n en {10,25,50,100}; Phi-4-14B, simple, n=10 logra el menor RMSE medio (21,587), apenas 0,020 mejor que el mismo modelo con n=50. La afirmación narrativa de que n=10 gana de forma consistente tiene una excepción en Mistral-specific, cuyo mejor promedio es n=100, y varias excepciones por cuestionario. El artículo aporta evidencia útil de control de respuestas de cuestionario y señales lingüísticas mediante prompting explícito, pero no demuestra personalidad interna, equivalencia humana ni fidelidad individual alta.

Research question

Can a training-free instruction, expressing the five Big Five traits through numerical values and descriptions of varying detail, proportionally control LLM responses, produce distinguishable and consistent signals in dialogue, and approximate human profiles?

Method

Evaluation in five blocks: induction of one trait in Alpaca-7B with MPI and baselines; correlation between ten assigned intensities and three questionnaires in three LLMs; 20-turn dialogues judged by three LLMs; similarity between the first nine turns and the tenth using cosine, BERTScore, and PersonaCLR; and reproduction of the IPIP-NEO-120 of 17 participants. An additional design factorial analysis generates 50 agents per combination of three models, three prompts, and four scales, normalizes to 100, and compares RMSE.

Sample: Alpaca-7B for the induction benchmark; three open models for scaling, dialogue, and consistency; ten levels of the target trait per combination in the correlational analysis; 20 turns per dialogue, ten from each agent, although how many dialogues is not published; 50 random agents per combination in the scale analysis; and 17 Korean NLP graduate students for human imitation. No number of runs, seeds, detailed demographic distribution, or sample size per dialogue condition is reported.

Findings

  • The simple prompt outperforms baselines on four of five MPI means, but not on neuroticism, where the neutral condition is highest.
  • The majority of correlations between indicated score and questionnaire score are high for Mistral and Phi-4; LLaMA contains several weak, non-significant, or one negative relationships.
  • LLM judges distinguish profiles in dialogue above chance, but the absolute improvement ranges from 2.1 to 14.1 points and lacks human evaluation and uncertainty.
  • PersonaCLR is at 0.789–0.828 and BERTScore at 0.480–0.537; the reported cosine is exactly 0.999 across the three models.
  • Imitation of 17 human profiles yields RMSE 1.785–1.822 versus an approximate chance of 2.0, a small advantage not statistically tested.
  • Phi-4-14B with simple prompt and n=10 has the lowest mean RMSE in Table 7, but practically ties with n=50 and not all subcases favor low scales or simple prompts.
  • Qualitative examples show stereotypical differences consistent with openness, agreeableness, and neuroticism, without independent systematic evaluation.

Limitations

  • The main validation reuses questionnaires of the same construct that the prompt explicitly names and describes; there is criterion contamination, demand characteristics, and the possibility of responding by semantic recognition.
  • There is no control condition separating obedience to numbers, knowledge of items, and behavioral expression of traits in tasks unrelated to the questionnaire.
  • Correlations are calculated with only ten levels per trait, without repetitions or intervals; many tests are published without correction for multiplicity and p=0.000 instead of bounded values.
  • The number of dialogues, profiles, repetitions, and evaluated units is not reported; nor are CIs, tests, inter-judge agreement, or a complete evaluation protocol.
  • Dialogue evaluators are LLMs and may share the same explicit stereotypes of the prompt; there is no blind human evaluation.
  • Comparing content of nine turns with a tenth mixes topic, vocabulary, memory, and style with personality. The cosine that produces 0.999 is not precisely defined, nor is PersonaCLR validated for this use in English.
  • The human benchmark has only 17 Korean NLP students, with no demographics, recruitment, consent, ethical approval, exclusions, or privacy analysis described.
  • The human chance of RMSE≈2.0 is presented as an approximation without procedure, distribution, interval, or test against the RMSE 1.785–1.822.
  • Exact checkpoint IDs, seeds, dates, hardware, number of runs, format errors, refusals, or code and data to reproduce the study are not published.
  • The name Mistral-25B is not reconciled with the cited reference of Mistral 7B; Table 5 additionally changes the models to Mistral-14B and Phi4-25B, inverting their declared sizes.
  • The conclusion that n=10 always wins contradicts Mistral-specific, whose lowest mean RMSE appears at n=100, and numerous cells per questionnaire.
  • The mean advantage of Phi-4 simple n=10 over n=50 is 0.020 normalized points, without a noise estimate that would allow attributing meaning to the difference.
  • Several facet descriptions are psychometrically imprecise or contain errors: straightforwardness is described as forgiveness, impulsiveness as emotional instability, and self-disciplinel and compilance appear.
  • The study assumes static traits, uses short conversations and prefixed topics, and does not evaluate longitudinal persistence, contextual change, safety, utility, or effects on users.

What the study does not establish

  • It does not demonstrate that the model possesses an internal, stable personality equivalent to human personality.
  • It does not demonstrate general proportional control outside of self-reports that repeat the prompt vocabulary.
  • It does not demonstrate high-fidelity individual imitation: the improvement over approximate chance is small and not tested.
  • It does not demonstrate psychological consistency from textual or stylistic similarity between turns.
  • It does not allow claiming that simple prompt or scale 10 are optimal for all models, traits, instruments, or tasks.
  • It does not validate clinical, educational, selection, social simulation, or human participant replacement uses.

Traceability

Scope: Full text

Version: arXiv:2508.06149v1 (8 August 2025)

Consulted source: https://arxiv.org/pdf/2508.06149

Review: Codex editorial review, 2026-07-14

Approval: Codex fidelity pass, 2026-07-14

English translation: approved, 2026-07-18

Models evaluated

  • Alpaca-7B
  • LLaMA3-8B (exact checkpoint not reported)
  • Mistral-25B as named by the paper (exact model unresolved; cited reference is Mistral 7B)
  • Phi-4-14B (exact checkpoint not reported)
  • GPT-4o-mini as dialogue evaluator (snapshot not reported)
  • Claude 3.5 Haiku as dialogue evaluator (snapshot not reported)
  • DeepSeek-Chat as dialogue evaluator (snapshot not reported)
  • xlm-roberta-base adaptation for PersonaCLR

Instruments and metrics

  • Machine Personality Inventory, reported as 1,000 items
  • Big Five Inventory
  • IPIP-NEO-120
  • NEO Five-Factor Inventory
  • Pearson correlation
  • LLM three-way trait identification
  • Cosine similarity
  • Sentence-BERT similarity
  • PersonaCLR adapted to English
  • Root mean squared error

Data used

  • Machine Personality Inventory from Jiang et al. (2023)
  • BFI, IPIP-NEO-120 and NEO-FFI questionnaire items
  • Generated 20-turn dialogues over ten predefined topics
  • IPIP-NEO-120 responses from 17 Korean NLP graduate students
  • Naro Utterance dataset used by the referenced PersonaCLR adaptation

Evidence and location

  • Definition of Big5-Scaler, scales, and three types of prompt: arXiv v1, pp. 3–4, sections 3.2–3.3
  • Models, parameters, and MPI induction design: arXiv v1, pp. 4–5, sections 4 and 4.1
  • Proportional design with three questionnaires and ten levels: arXiv v1, p. 5, section 4.2
  • Dialogues, LLM judges, and consistency metric: arXiv v1, pp. 5–6, sections 4.3–4.4
  • Human sample of 17 participants and RMSE: arXiv v1, p. 6, section 4.5
  • MPI scores, correlations, and per-trait exceptions: arXiv v1, pp. 6–7, sections 5.1–5.2 and Tables 1–2
  • Profile identification in dialogue: arXiv v1, pp. 6–7, section 5.3 and Table 3
  • Cosine, BERTScore, and PersonaCLR: arXiv v1, pp. 7–8, section 5.4 and Table 4
  • Human imitation and inconsistent model labels: arXiv v1, pp. 7–8, section 5.5 and Table 5
  • Analysis of 50 agents per combination and claims about scale: arXiv v1, p. 8 and p. 18, section 6 and Table 7
  • Limitations acknowledged by the authors: arXiv v1, p. 9, section 8
  • Exact content and errors of the prompts: arXiv v1, pp. 12–16, Appendices A–B and Figures 2
  • Curves and qualitative dialogue examples: arXiv v1, pp. 17 and 19, Figures 3–5