Linear Personality Probing and Steering in LLMs: A Big Five Study

Trait induction and control2025arXivApproved editorial review

Authors: Michel Frising, Daniel Balcells

Keywords: Large Language Models, Personality, Big Five, Persona, Personality Control

Source: Open primary source (opens in a new tab)

2
Authors
20
Findings
44
Limitations
11
Evidence

Editorial summary

English

This preprint asks whether linear directions in Llama 3.3 70B activations can represent and modify the Big Five traits. The same model answers the 50-item IPIP questionnaire as 406 fictional characters and explains every answer. The authors sum ten items per trait to create five scores and concatenate the explanations into character descriptions. They then pair each description with ten Alpaca instructions and collect every-layer activations at the last input token, the mean input tokens, and the mean generated tokens. For each trait score, they average activations over all characters sharing that score and fit a linear regression per trait and layer, with SVD axes as a comparison. Supervised directions are nearly orthogonal across traits and separate positive from negative adjectives when the adjective itself is inserted literally into the system prompt. This is compatible with explicit semantic encoding, but it does not validate personality detection: labels and text come from the same model, the explanations restate item content and polarity, no character split, cross-validation, or independent human labels are documented, and the adjective test lacks a published list, sample sizes, numeric AUCs, lexical controls, negation controls, or alternative templates. The causal evaluation is narrower than the title suggests. A direction is injected at the final input token across all layers, with |alpha| capped at 0.4 because larger values produce incoherent text. The reported quantitative task tests Extraversion only: the model selects exactly five of ten statements, five extraverted and five introverted. Five statements were already used to build the directions and five came from an extended inventory. The mean-input regression direction monotonically changes selection without extra context; last-token and SVD directions fail and the mean-output direction is less stable. No repetitions, intervals, or tests are reported. Adding a character description eliminates the effect. Across ten open-ended prompts, qualitative inspection finds minimal response and uses no blinded raters, rubric, metric, or statistical analysis. Percentiles from 300,000 human responses only show that synthetic scores lie in a human range; they do not test agreement with human ratings of each character. Two examples in which an entity called ChatGPT-5 identifies Tony Soprano and Lady Mary illustrate identity leakage, but they are not a systematic test and lack a reproducible prompt or snapshot. The defensible conclusion is narrow: within one Llama model, directions fitted to synthetic questionnaire-derived descriptions correlate with explicitly induced trait language, and one direction can shift a constrained Extraversion task in a narrow context. The work does not establish latent personality, a causal trait circuit, generalization beyond the model or characters used, control of all five traits, or robust open-ended steering. The PDF declares code and data, but the GitHub repository returned 404 and the Hugging Face dataset returned 401 on 15 July 2026, so both are recorded as declared but inaccessible.

Español

Este preprint estudia si direcciones lineales de las activaciones de Llama 3.3 70B pueden representar y modificar los cinco rasgos Big Five. El propio modelo responde, como 406 personajes ficticios, los 50 ítems IPIP y explica cada respuesta. Los autores suman los diez ítems de cada rasgo para obtener cinco puntuaciones y concatenan las explicaciones como descripciones de personaje. Después presentan cada descripción junto a diez instrucciones de Alpaca y extraen activaciones en todas las capas: último token de entrada, media de tokens de entrada y media de tokens generados. Para cada puntuación de rasgo, promedian las activaciones de todos los personajes que comparten esa puntuación y ajustan una regresión lineal por rasgo y capa; también comparan ejes SVD. Las direcciones supervisadas quedan casi ortogonales entre rasgos y separan adjetivos positivos y negativos cuando el propio adjetivo se inserta literalmente en el system prompt. Este resultado es compatible con codificación semántica explícita, pero no valida detección de personalidad: las etiquetas y los textos proceden del mismo modelo, las explicaciones reformulan el contenido y la polaridad de los ítems, no se documenta partición de personajes, validación cruzada ni etiquetas humanas independientes, y la prueba de adjetivos carece de lista, tamaños muestrales, AUC numéricas, controles léxicos, negación o plantillas alternativas. La evaluación causal es más estrecha que el título. Se inyecta una dirección en el último token de entrada a través de todas las capas, con |alpha| hasta 0,4 porque valores mayores producen texto incoherente. La tarea cuantitativa publicada solo examina Extraversion: el modelo debe elegir exactamente cinco de diez frases, cinco extravertidas y cinco introvertidas. Cinco frases ya pertenecen al cuestionario usado para construir las direcciones y cinco se toman de un inventario ampliado. La dirección de regresión sobre la media de entrada cambia monotónicamente la selección sin descripción adicional; las basadas en último token o SVD fallan y la media de salida es menos estable. No se informan repeticiones, intervalos ni tests. Cuando se añade una descripción de personaje, el efecto desaparece. En diez prompts abiertos, la inspección cualitativa encuentra una respuesta mínima y no usa jueces ciegos, rúbrica, métrica o análisis estadístico. Los percentiles de 300.000 respuestas humanas solo muestran que las puntuaciones sintéticas caen en un rango humano; no comprueban que coincidan con valoraciones humanas de cada personaje. Dos ejemplos en que una entidad llamada ChatGPT-5 reconoce a Tony Soprano y Lady Mary ilustran que las descripciones filtran identidad, pero no constituyen una prueba sistemática y no incluyen prompt o snapshot reproducible. La conclusión defendible es limitada: dentro de un único Llama, direcciones ajustadas a descripciones sintéticas derivadas del cuestionario correlacionan con lenguaje de rasgos explícitamente inducido y una dirección puede desplazar una tarea forzada de Extraversion en un contexto estrecho. El trabajo no demuestra una personalidad latente, un circuito causal de rasgo, generalización fuera del modelo o personajes usados, control de los cinco rasgos ni dirección robusta de conversación abierta. El PDF declara código y datos, pero el repositorio GitHub devolvía 404 y el dataset de Hugging Face 401 el 15 de julio de 2026, de modo que ambos artefactos se registran como declarados pero inaccesibles.

Research question

Can linear directions learned in the activations of Llama 3.3 70B from synthetic Big Five profiles detect language associated with each trait and modify the model's response?

Method

Llama 3.3 70B answers the 50 IPIP items as 406 characters and generates one explanation per item. Ten responses per trait are aggregated and the explanations form a description. Each description is combined with ten Alpaca instructions, producing 4,060 activation contexts. In each layer the last input token, input mean, and output mean are extracted. For each score value the activations of characters with that value are averaged and least squares is fitted; SVD serves as comparison. The directions are tested with adjectives inserted in the prompt and injected at the last input token of all layers. The published quantitative control uses ten Extraversion sentences and the open analysis qualitatively inspects ten instructions.

Sample: The analytical sample is 406 fictional characters, not persons or independent models. Each character generates 50 responses and explanations, for 20,300 item observations, and is combined with ten instructions, for 4,060 contexts. Regression then reduces the activations to averages per score value. Only one Llama checkpoint is studied; the quantitative steering task uses a single set of ten Extraversion sentences and no repetition by alpha is reported.

Findings

  • The verified current source is arXiv:2512.17639v2, reviewed on 15 January 2026, with 29 pages and six figures.
  • The 29 pages were rendered and visually inspected.
  • The current PDF matches byte for byte with the cached copy and has SHA-256 2399356785fe311d88807c48dd52c8fa90caf727001a6de59c9ffe61e24c9aa0.
  • Llama 3.3 70B generates 406 profiles from 50 IPIP responses per character.
  • The response explanations are reused as descriptions that condition the activations.
  • Activations are collected across 4,060 combinations of character and instruction, in all layers and three token positions.
  • The regression is fitted to activations averaged across characters with the same trait score.
  • The regression directions of different traits are approximately orthogonal in the illustrated layer.
  • The SVD axes are nearly orthogonal to the supervised directions and do not produce systematic steering.
  • Positive and negative adjectives separate better in middle and late layers when the adjective is included literally in the prompt.
  • The injection is applied to the last input token in all layers and is empirically limited to |alpha| less than or equal to 0.4.
  • The input mean direction monotonically changes a forced selection of Extraversion sentences without additional description.
  • The output mean direction is less reliable; last token and SVD do not achieve a complete or systematic change.
  • When an explicit character description is added, steering has no observable effect on the forced task.
  • The open evaluation reports a minimal response to steering through qualitative inspection.
  • Five of the ten sentences in the forced task already appear in the questionnaire that built the direction.
  • The synthetic scores fall within human percentiles, without checking character-by-character agreement.
  • The appendix shows two descriptions that an entity named ChatGPT-5 assigns to the expected character with 95% declared confidence.
  • The paper declares code on GitHub and data on Hugging Face.
  • On 15 July 2026 the repository returned HTTP 404 and the dataset HTTP 401, so they could not be audited or reproduced.

Limitations

  • The trait labels, explanations, and activations come from the same Llama 3.3 70B.
  • The explanation of each item re-expresses its semantics, response, and polarity, creating direct label leakage into the description.
  • The study measures activations induced by explicit descriptions, not the model's behavior without a prompt.
  • No training and test partition by characters is documented.
  • No cross-validation, bootstrap, or out-of-sample evaluation is described for the regression.
  • There are no independent human labels for the scores of the 406 characters.
  • The synthetic scores are not compared with the empirical ratings of characters that the OpenPsychometrics source claims to offer.
  • Being within a human range does not demonstrate realism, validity, or profile agreement.
  • Averaging by score reduces the analytical unit to bins and may hide variation across characters.
  • It is not reported how many unique score values feed each regression.
  • The scores of ten items only allow at most 41 discrete sums before considering unobserved values.
  • The equation assumes normal errors without reported diagnosis or justification.
  • No coefficients, R-squared, prediction error, or residuals of the regressions are published.
  • Orthogonality among weights does not prove that each axis corresponds causally to a psychological trait.
  • The adjective evaluation literally introduces the descriptor into the system prompt.
  • ROC may detect lexical semantics or instruction following, not generalizable personality.
  • The list of adjectives, their provenance, balance, or number per trait is not published.
  • No exact AUCs, intervals, tests, or correction for multiple layers and traits are reported.
  • There are no controls with synonyms, antonyms, negation, neutral words, contradictory sentences, or alternative templates.
  • There is no testing on held-out characters, other models, base models, or languages.
  • The published quantitative intervention evaluates only Extraversion, not the five traits.
  • Half of the sentences in the forced task were seen during the construction of the directions.
  • The choose-five-of-ten format forces a mechanical trade-off between positive and negative options.
  • The curves appear to be a single trajectory per alpha without repetitions or uncertainty.
  • No seed, temperature, top-p, max tokens, sampling, or treatment of invalid responses is reported.
  • The normalization of each direction is not reported, so the alpha scale may not be comparable across methods, traits, or layers.
  • Larger values of alpha produce gibberish, a sign of general perturbation in addition to possible trait control.
  • Injection in all layers prevents attributing the effect to a specific layer.
  • The explicit description cancels the effect, limiting robustness to real context.
  • The open evaluation does not define in advance what change counts as Extraversion or another trait.
  • There are no blind human judges, inter-rater agreement, LIWC, LLM-as-judge, metric, or statistical test in open tasks.
  • The open examples change invented autobiographical content in addition to style, making it difficult to attribute the effect to the trait.
  • One of the ten instructions asks for the lyrics of Yesterday, a task with likely copyright rejection or restriction that is not analyzed as a confounder.
  • The two identity analyses attributed to ChatGPT-5 are selected examples, not a systematic evaluation of leakage.
  • It is not identified which product, snapshot, or prompt corresponds to ChatGPT-5.
  • No exact revision of the Llama checkpoint, libraries, hardware, quantization, or numerical precision is reported.
  • No complete numerical tables for ROC or steering curves are published.
  • There is no evaluation of capability, factuality, safety, or coherence beyond observing gibberish.
  • The declared code was not publicly available at the indicated URL during the audit.
  • The declared dataset required authorization or was not publicly available during the audit.
  • Without accessible artifacts, exclusions, parsing, averages, regressions, or graphs cannot be verified.
  • The synthetic design does not justify claiming that the profiles are grounded in validated human psychometrics.
  • Applying a human scale to generated text does not establish construct equivalence in an LLM.
  • The study is a preprint with no evidence of peer review indicated in the source.

What the study does not establish

  • It does not establish that Llama 3.3 70B possesses a stable or human personality.
  • It does not demonstrate that the directions are causal circuits of personality.
  • It does not distinguish a trait representation from the explicit semantics of the prompt.
  • It does not demonstrate out-of-sample prediction on held-out characters.
  • It does not validate the character scores against human judgments.
  • It does not demonstrate control of Agreeableness, Conscientiousness, Emotional Stability, or Openness.
  • It does not demonstrate robust steering when character context exists.
  • It does not demonstrate reliable control of open generation.
  • It does not generalize to other models, checkpoints, languages, or interfaces.
  • It does not show that a direction is better than prompting under comparable conditions.
  • It does not allow reproducing the result with the currently accessible artifacts.
  • It does not evaluate safety or side effects of steering.

Traceability

Scope: Full text

Version: arXiv:2512.17639v2, submitted 19 December 2025 and revised 15 January 2026, 29 pages and 6 figures

Consulted source: https://arxiv.org/abs/2512.17639

Review: Codex complete bilingual full-text fidelity pass, current arXiv-version reconciliation, all-page PDF visual inspection, declared-code and dataset availability checks, experimental-count reconstruction, label-leakage and validation-design audit, psychometric and causal-claim assessment; summaries written from the full paper rather than abstract keywords, 2026-07-15

Approval: Codex fidelity pass, 2026-07-15

English translation: approved, 2026-07-18

Models evaluated

  • meta-llama/Llama-3.3-70B-Instruct for questionnaire generation, character descriptions, activation collection, probing and steering; exact checkpoint revision, runtime and hardware not reported
  • ChatGPT-5, as named by the appendix, for two character-identity guesses; provider, snapshot, prompt and run date not reported

Instruments and metrics

  • 50-item IPIP Big Five Factor Markers, ten items each for Extraversion, Emotional Stability, Agreeableness, Conscientiousness and Openness
  • Five-point Likert responses plus free-text explanation generated in character
  • Linear least-squares directions fitted to score-grouped mean activations
  • Singular-value decomposition axes as unsupervised comparison
  • Adjective-polarity projection and ROC visualization
  • Ten-item forced-choice Extraversion task, with five items seen during direction construction and five reported as unseen
  • Qualitative inspection of ten open-ended Alpaca instructions

Data used

  • 406 fictional characters selected from OpenPsychometrics Which Character Are You lists
  • 20,300 model-generated questionnaire item responses and explanations
  • 4,060 character-description and Alpaca-instruction activation contexts
  • Human percentile reference derived from 300,000 OpenPsychometrics Big Five responses
  • Ten selected instructions attributed to the tatsu-lab/alpaca dataset
  • Declared plastic-labs/personality-steering Hugging Face dataset, inaccessible with HTTP 401 on 15 July 2026

Evidence and location

  • Version, date, length, model, and declared contribution: arXiv:2512.17639v2, title page, abstract and arXiv metadata checked 15 July 2026
  • Generation of 406 profiles and 50 responses per character: Sections 2.1 and 3.1, pages 3 and 5; Listing 1
  • Ten instructions, three positions, and activations in all layers: Section 2.2, pages 3–4; Appendix D, page 15
  • Regression on means of characters with equal score: Section 2.3, equation 1, page 4
  • Adjectives and ROC: Section 3.2.2 and Figure 4, page 6; Listing 4, page 13
  • Injection method and empirical alpha threshold: Section 3.3.1, equation 2, page 7
  • Forced task, overlap of five items, and results: Section 3.3.2, Figure 5 and Listing 3, pages 7–8; Figure 6, page 15
  • Null effect with description and minimal signal in open task: Sections 3.3.2–3.3.3, Figures 5–6 and Appendix I, pages 7–8 and 27–29
  • Human percentiles and identity leakage analysis: Section 3.1 and Figure 2, page 5; Appendices E–H, pages 16–24
  • Declared code and dataset but inaccessible: Paper page 1 footnotes; GitHub page/API HTTP 404 and Hugging Face page/API HTTP 401 checked 15 July 2026; reports/verification/article-180-artifact-and-design-audit.json
  • Complete visual inspection: All 29 pages of arXiv:2512.17639v2 rendered and visually inspected on 15 July 2026