Synthetic Personality Review

Deterministic AI Agent Personality Expression through Standard Psychological Diagnostics

Evaluation and psychometric validity2025allora.networkApproved editorial review

Authors: J. M. Diederik Kruijssen, Nicholas Emmons

Keywords: Machine Learning, Artificial Intelligence, Computers and Society, Human-Computer Interaction

Source: Open primary source (opens in a new tab)

2
Authors
17
Findings
40
Limitations
22
Evidence

Editorial summary

English

This Allora Foundation paper tests whether four GPT models follow personality templates explicitly placed in the system prompt. A GPT-4o 'character builder' creates ten agents with 1–5 Big Five values, an MBTI type, detailed trait descriptions, style, backstory, and trading behavior. GPT-4o-mini (2024-07-18), GPT-4o (2024-08-06), o1 (2024-12-17), and o3-mini (2025-01-31) then answer one 50-item Big Five and one 70-question MBTI sequence per condition. Some conditions require a short motivation; GPT-4o is also compared with a version fine-tuned on 130 GPT-4o-generated pairs for an 'unhinged' crypto style. The aggregate score, an average of 16 heterogeneous normalized metrics, is .63 for 4o-mini, .78 for 4o and o3-mini, and .79 for o1; with motivations it becomes .70, .72, .74, and .78. Fine-tuned GPT-4o scores .76 without motivations and .72 with them. These differences have no intervals or inferential tests: the reported standard deviation is dispersion across metrics, not uncertainty. The sample does not uniformly span personality space: only five of sixteen MBTI types appear, four agents are ENTP, eight of ten are intuitive, and two agents share exactly the same five scores and ISTJ type. The clearest result is that proprietary models can obey an instruction that directly supplies a profile and then produce questionnaire answers compatible with it. Determinism is not tested: there are no repeated runs of the same condition, temperature, top-p, seed, or temporal-stability measurements. The design also does not separate prompt memory, semantic recognition of items, and behavior outside the test. The 'holistic reasoning' inference rests on individual responses not falling on a perfect diagonal; the same pattern is compatible with noise, item ambiguity, or imperfect instruction following, and there is no human baseline or causal test. In motivated o1, openness fails markedly: target mean 3.90 (SD 1.29) versus tested mean 4.67 (SD .42). There is also a mathematical error: for MBTI the paper defines Cohen's observed agreement as TP/n and omits true negatives, so its kappa is not standard binary kappa. The claim that fine-tuning only changes style is based on a single-run comparison and selected examples; the paper attributes one difference to 'Poisson noise' without a probability model or test. No code, full 130-pair dataset, responses, API configuration, or per-agent results are published. ADI assigns a DOI and hosts the paper, but it was created by Allora, says it initially focuses on work by its own team, and describes external peer-reviewed contributions as a future goal; both authors are at Allora Foundation. This is internally published research, not independent validation. The paper does not test users, trust, education, health care, therapy, safety, or trading behavior. The defensible conclusion is narrow: under prompts that directly disclose target profiles, these GPT snapshots generate questionnaire patterns correlated with those targets, with a persistent bias toward high openness.

Español

Este artículo de Allora Foundation evalúa si cuatro modelos GPT siguen plantillas de personalidad incluidas explícitamente en el system prompt. Un GPT-4o «character builder» genera diez agentes con valores Big Five de 1 a 5, un tipo MBTI, descripciones de cada rasgo, estilo, historia y conducta de trading. Después, GPT-4o-mini (2024-07-18), GPT-4o (2024-08-06), o1 (2024-12-17) y o3-mini (2025-01-31) responden una vez, pregunta a pregunta, a 50 ítems Big Five y 70 preguntas MBTI. Algunas condiciones exigen una breve justificación; GPT-4o también se compara con una versión afinada en 130 pares generados por GPT-4o para producir un estilo cripto «unhinged». La cifra agregada, que promedia 16 métricas normalizadas heterogéneas, es 0,63 para 4o-mini, 0,78 para 4o y o3-mini, y 0,79 para o1; con justificación pasa a 0,70, 0,72, 0,74 y 0,78, respectivamente. El GPT-4o afinado obtiene 0,76 sin justificación y 0,72 con ella. Estas diferencias no tienen intervalos ni tests inferenciales: la desviación publicada es dispersión entre métricas y no incertidumbre. La muestra tampoco representa uniformemente el espacio de personalidad: solo aparecen 5 de 16 tipos MBTI, cuatro agentes son ENTP, ocho de diez son intuitivos y dos agentes comparten exactamente los mismos cinco scores y tipo ISTJ. El resultado más claro es que modelos propietarios pueden obedecer una instrucción que les entrega el perfil y luego producir respuestas compatibles con él. No se prueba determinismo: no hay réplicas de una misma condición, temperatura, top-p, seed ni estabilidad temporal. Tampoco se separa memoria del prompt, reconocimiento semántico de los ítems y expresión conductual fuera del test. La inferencia de «razonamiento holístico» se apoya en que las respuestas individuales no caen en la diagonal perfecta; esa variación también es compatible con ruido, ambigüedad de ítems o cumplimiento imperfecto, y no hay baseline humano ni análisis causal. En la condición o1 con justificación, apertura falla de forma marcada: media objetivo 3,90 (SD 1,29) frente a 4,67 (SD 0,42). Hay además un error matemático: para MBTI el paper define el acuerdo observado de Cohen κ como TP/n y omite verdaderos negativos, por lo que ese κ no es el estándar binario. La afirmación de que el fine-tuning solo cambia el estilo se basa en una comparación de un único run y ejemplos escogidos; el texto incluso atribuye una diferencia a «Poisson noise» sin modelo probabilístico ni test. No se publica código, el dataset completo de 130 pares, respuestas, configuración API ni resultados por agente. ADI asigna DOI y ofrece el PDF, pero fue creada por Allora, declara centrarse inicialmente en trabajos del propio equipo y presenta la revisión por pares externa como una aspiración futura; los autores pertenecen a Allora Foundation. La evidencia debe tratarse como investigación interna publicada, no como validación independiente. No se evalúan usuarios, confianza, educación, salud, terapia, seguridad ni conducta de trading. La conclusión defendible es estrecha: bajo prompts que revelan directamente los objetivos, estos snapshots GPT generan patrones de cuestionario correlacionados con ellos, con un sesgo persistente hacia alta apertura.

Research question

To what extent do four GPT snapshots answer Big Five and MBTI questionnaires congruently with numerical and narrative profiles included in their system prompt, and do those results change when justifications are requested or the style of GPT-4o is fine-tuned?

Method

GPT-4o generates ten templates with five Big Five scores, a compatible MBTI type and behavioral descriptions. Each template is inserted into the system prompt of four models. Per condition, the agent sequentially answers 50 Big Five Likert items and 70 MBTI choices, with or without a justification. MAE, RMSE, Pearson, Spearman and κ are calculated for Big Five, and F1, accuracy and a binary κ formulated incorrectly for MBTI, both on final scores and on responses. The 16 normalized metrics are averaged. GPT-4o is additionally compared with a style fine-tune trained on 130 synthetic pairs. There are no replicates per condition.

Sample: Ten synthetic agents, four base models and ten experimental conditions. Each condition uses the same ten profiles and a single administration of 50 Big Five questions and 70 MBTI per agent. At the test level there is n=10 per dimension; at the response level items nested within those same ten agents are pooled. Only five MBTI types are covered and the distribution is heavily skewed.

Findings

  • The aggregated mean of 16 metrics is 0.63 for 4o-mini, 0.78 for 4o, 0.78 for o3-mini and 0.79 for o1.
  • When motivation is required, the means change to 0.70, 0.72, 0.74 and 0.78, respectively.
  • Fine-tuned GPT-4o obtains 0.76 without motivation and 0.72 with it, compared to 0.78 and 0.72 for the non-fine-tuned GPT-4o.
  • The differences between 0.78 and 0.79 include no hypothesis test, interval or repetition that would allow ordering models with uncertainty.
  • For GPT-4o without motivation, Big Five MAE is 0.44 in extraversion, 0.61 in agreeableness, 0.13 in conscientiousness, 0.04 in neuroticism and 0.80 in openness.
  • For o1 without motivation, the Big Five MAEs are 0.38, 0.72, 0.20, 0.20 and 0.78.
  • Openness is the systematic failure: in motivated o1 the target mean 3.90 becomes 4.67 and the SD drops from 1.29 to 0.42.
  • The authors observe more dispersion at the item level than in the final score and interpret it as personality-based reasoning.
  • The non-diagonal matrices only show that individual responses are not perfect copies of the target score; they do not identify the mechanism that produced the variation.
  • The motivation requirement improves the aggregate of 4o-mini, reduces those of 4o and o3-mini and barely changes o1.
  • The justification examples are usually semantically compatible with the number, but they were selected by the authors and not evaluated blindly.
  • The fine-tune visibly changes the tone of five examples toward more cryptic and aggressive responses; there is no formal measure of style over the complete set.
  • Four of ten agents are ENTP, two ENFP, two ISTJ, one INTJ and one ENFJ; eleven MBTI types do not appear.
  • Eight of ten agents are N and no profile represents uniformly low scores across the five traits.
  • Agent 2 and Agent 8 share exactly Big Five 2/4/5/2/3 and ISTJ type, reducing the effective diversity of the targets.
  • The MBTI κ formula uses TP/n as observed agreement and a chance term that is only positive; it omits TN and does not compute the conventional binary κ.
  • No complete responses, MBTI results per condition, per-agent data, API configuration or code to recalculate the figures are published.

Limitations

  • There are no multiple generations of the same agent-model-test combination; the assumed determinism across runs is not measured.
  • Temperature, top-p, seed, max tokens, retries, API date and other sampling parameters are not reported.
  • The models are proprietary services and the snapshots may become unavailable, with no complete outputs for preservation.
  • The system prompt directly contains the five Big Five values, the MBTI type and detailed explanations of how each level should behave.
  • The prompt orders careful adherence to the personality; the task measures instruction following with strong target leakage.
  • There is no baseline without profile, with hidden profile, with permuted labels or with a narrative without numerical values.
  • Big Five and MBTI are not independent validations because both targets are included simultaneously in the template.
  • Assigning each item the global trait score as the true response assumes that all items should equal the latent, a non-psychometric simplification.
  • Responses are nested within ten agents, but response-level metrics pool them without a multilevel model.
  • At the test level there are only ten points per trait, with repeated values and restricted range for several dimensions.
  • There is no comparison with test-retest reliability, human variability or human distribution of responses per item.
  • The non-diagonal pattern does not distinguish holistic reasoning from sampling noise, ambiguity, response bias or imperfect compliance.
  • The claim about intelligence and reasoning compares four models that differ in multiple aspects without factorial manipulation of those capabilities.
  • The labels "low/high intelligence" and "low/intermediate/high reasoning" are assigned qualitatively and are not measured in the experiment.
  • The paper averages 16 metrics with different scales, dependencies and meanings, giving them the same weight without justification.
  • The standard deviation of the aggregate is variation between metrics and dimensions, not standard error, interval or variability across runs.
  • Figure 7 uses percentiles among only four or five dimensions; the caption itself acknowledges that the bars are not uncertainty.
  • The Cohen κ formula for MBTI is incorrect because observed agreement must include TP and TN and chance must include both classes.
  • It is not specified whether F1 is per class, macro, micro or weighted; its value depends on which MBTI pole is treated as positive.
  • Unweighted Cohen κ treats adjacent and extreme Likert errors equally; it is not justified against an ordinal κ.
  • There is no correction for the numerous comparisons of models, conditions, tests, scales, traits and metrics.
  • The expressions "not statistically significant" and "Poisson noise" appear without test, p, interval, likelihood or justification of a Poisson distribution for bounded ordinal scores.
  • Only five of sixteen MBTI types are present and four profiles are ENTP; the claim of broad coverage does not hold for MBTI.
  • Means close to 3 do not demonstrate a flat or representative distribution of the five traits.
  • Two profiles have identical Big Five and MBTI targets; the study does not separate the effect of their different narratives.
  • The ten templates were generated by a single GPT-4o under unspecified "controlled randomization" and then selected without seed or record of the process.
  • The exact source or validation of the 50 Big Five items and the 70 MBTI questions is not published.
  • The title refers to psychological diagnoses, but trait questionnaires and MBTI do not constitute clinical diagnosis.
  • The paper itself acknowledges MBTI as pseudoscientific; using it does not demonstrate psychometric generalization.
  • The exact fine-tune, hyperparameters, epochs, loss, validation and 120 of the 130 training examples are not published.
  • All fine-tuning pairs were generated by GPT-4o, with no comparison to human data or a control of equal volume and different style.
  • Independence between style and personality is inferred from a single fine-tune of a single model and a single administration per condition.
  • Style is judged with five selected examples, without annotators, blinding, rubric, agreement or quantitative metric.
  • No human-perceived personality, multi-turn persistence outside the test or behavior in tasks or tool calls is evaluated.
  • Trading behavior is in the templates, but the paper declares that it does not test it.
  • There is no evaluation of utility, trust, engagement, safety, manipulation, education, therapy or health despite the proposed applications.
  • There is no code, complete dataset, outputs, data tables, artifact license or reproduction instructions.
  • The authors belong to Allora Foundation and the journal ADI was created by Allora to initially publish work by the team itself.
  • The presentation of ADI describes external selected peer-reviewed contributions as a future aspiration, without documenting independent review of this article.
  • There is no formal declaration of conflict, funding, data availability, ethical approval or sponsor role in the PDF.

What the study does not establish

  • It does not demonstrate determinism or reproducibility across runs.
  • It does not demonstrate an internal, stable or conscious personality in any model.
  • It does not show emergent personality: the targets and their descriptions are provided directly.
  • It does not validate a psychometric instrument for AI agents.
  • It does not prove that responses arise from holistic reasoning.
  • It does not causally demonstrate that intelligence or reasoning explain the differences between models.
  • It does not establish that o1 outperforms GPT-4o or o3-mini with significance or stability.
  • It does not demonstrate general independence between communication style and expressed personality.
  • It does not show broad coverage of the MBTI or Big Five space.
  • It does not demonstrate that personality persists outside the questionnaires or when prioritizing another task.
  • It does not evaluate trading behavior, tool use or real decisions.
  • It does not demonstrate that users perceive the profiles as natural, distinct, reliable or attractive.
  • It does not validate applications in education, health, therapy or customer service.
  • It does not demonstrate safety or absence of manipulation or emotional dependence.
  • It does not allow reproducing the complete results with the published materials.
  • It does not constitute an independent editorial validation by Allora Foundation.

Traceability

Scope: Full text

Version: Allora Decentralized Intelligence 2, 15–39, final publisher PDF; DOI 10.70235/allora.0x20015

Consulted source: https://research.assets.allora.network/allora.0x20015.pdf

Review: Codex full-text, bilingual-fidelity, visual, publisher-independence, prompt-leakage, psychometric-construct, test-retest, sampling, nested-unit, metric-formula, aggregation, inferential-statistics, model-version, fine-tuning, reproducibility, human-evaluation, product-claim, safety, ethics and evidence-level audit, 2026-07-15

Approval: Codex fidelity pass, 2026-07-15

English translation: approved, 2026-07-18

Models evaluated

  • GPT-4o-mini-2024-07-18
  • GPT-4o-2024-08-06
  • o1-2024-12-17
  • o3-mini-2025-01-31
  • GPT-4o character-builder agent
  • Fine-tuned GPT-4o variant with exact fine-tune identifier not reported

Instruments and metrics

  • 50-item Big Five questionnaire with 1–5 Likert responses and reverse scoring
  • 70-question binary MBTI questionnaire
  • MAE and RMSE
  • Pearson and Spearman correlations
  • Cohen kappa for Big Five response categories
  • F1 and accuracy for MBTI
  • Non-standard, incorrect binary Cohen kappa formula for MBTI
  • Aggregate mean of 16 normalized metrics
  • Qualitative inspection of selected motivations
  • Kernel-density visualization of ten o1-motivated scores per trait

Data used

  • Ten synthetic personality templates generated by GPT-4o
  • One Big Five response sequence and one MBTI response sequence per agent-model-condition
  • 130 GPT-4o-generated prompt-response pairs for communication-style fine-tuning; only ten are printed

Evidence and location

  • Identity, DOI, affiliation and complete abstract: ADI final PDF p. 1 (journal p. 15) and Allora article record
  • Exact models and snapshots: PDF p. 2 (journal p. 16), section 2.1
  • Template with scores, MBTI and behavioral descriptions: PDF pp. 2, 18–22 (journal pp. 16, 32–36), sections 2.1 and Appendix A
  • Big Five protocols and scoring: PDF p. 3 (journal p. 17), section 2.2.1 and Equations 1–2
  • MBTI protocol and scoring: PDF p. 4 (journal p. 18), section 2.2.2 and Equations 3–4
  • Ten profiles, duplication and MBTI coverage: PDF p. 5 (journal p. 19), Table 1
  • Ten conditions and aggregated scores: PDF p. 6 (journal p. 20), Table 2
  • Metric definitions and error in MBTI κ: PDF pp. 5–6 (journal pp. 19–20), Equations 5–6
  • Test and response unit of analysis: PDF p. 6 (journal p. 20), section 3.1
  • Big Five results per model: PDF pp. 7–8 (journal pp. 21–22), Figures 1–2 and section 3.2
  • Results with motivation and reasoning inference: PDF pp. 9–11 (journal pp. 23–25), section 3.3 and Figures 3–4
  • Fine-tuning and selected examples: PDF pp. 12–13 (journal pp. 26–27), section 3.4 and Figures 5–6
  • Poisson/significance claim without test: PDF p. 13 (journal p. 27), end of section 3.4
  • Normalization and aggregation of 16 metrics: PDF p. 14 (journal p. 28), section 3.5, Equation 7 and Figure 7
  • Bars are dispersion between dimensions, not uncertainty: PDF p. 14 (journal p. 28), Figure 7 caption
  • Systematic openness bias: PDF p. 15 (journal p. 29), Figure 8 and section 3.5
  • Acknowledged limitations and application claims: PDF pp. 16–17 (journal pp. 30–31), section 4
  • Complete questions of both tests: PDF pp. 23–25 (journal pp. 37–39), Appendix B
  • Fine-tune of 130 synthetic pairs and only ten published: PDF p. 25 (journal p. 39), Appendix C and Table C.1
  • Visual inspection: All 25 pages of the final ADI PDF rendered and visually inspected, including eight figures, three main/appendix tables and equations; checked 15 Jul 2026
  • Absence of linked reproducible artifacts: ADI article page, final PDF and targeted GitHub searches by exact title, distinctive prompt and DOI; checked 15 Jul 2026
  • Editorial model and lack of documented independent validation: Allora blog introducing ADI: initially focused on Allora Labs research and external selected peer-reviewed contributions described as a future pathway; checked 15 Jul 2026