Synthetic Personality Review

Do LLMs Possess a Personality? Making the MBTI Test an Amazing Evaluation for Large Language Models

Evaluation and psychometric validity2023arXivApproved editorial review

Authors: Keyu Pan, Yawen Zeng

Keywords: Large Language Models, Personality Assessment, Myers-Briggs Type Indicator (MBTI), Prompt Engineering, Artificial Intelligence

Source: Open primary source (opens in a new tab)

2
Authors
10
Findings
21
Limitations
14
Evidence

Editorial summary

English

This preprint explores whether a 93-item forced-choice MBTI questionnaire can describe response patterns across six LLMs, whether prompts alter those patterns, and whether continued training on different corpora changes the resulting label. The procedure counts E/I, S/N, T/F, and J/P responses and turns each pairwise majority into one of 16 types. It produces different labels, for example, ENTJ for ChatGPT, INTJ for GPT-4, ISTJ for BLOOM-7B, and INFJ for OpenLLaMA-7B-v2, and shows that an explicit prompt shifts ChatGPT from ENTJ to INFP while BLOOM and Baichuan change little. Continued training of BLOOM and OpenLLaMA on Chinese Wikipedia, question-answer, or APE210K data also changes some counts and labels. These are descriptive findings about questionnaire responses and intervention sensitivity, not evidence that models possess personality. Classification is highly fragile: several letters depend on ties or one-to-two-item margins; the code systematically breaks ties toward I/N/F/P; GPT-4 refusals are excluded; and there are no replications, uncertainty estimates, option-order controls, or validation against actual capability. The authors acknowledge MBTI's weak psychometrics but then infer reasoning and planning from T and J without external evidence. The defensible contribution is an early demonstration that prompts, training data, language, and scoring alter an anthropomorphic profile; MBTI is not validated as an LLM evaluation metric.

Español

Este preprint explora si un cuestionario MBTI de 93 elecciones forzadas puede describir patrones de respuesta de seis LLM, si esos patrones cambian con prompts y si el entrenamiento continuado con distintos corpus altera la etiqueta final. El procedimiento cuenta respuestas E/I, S/N, T/F y J/P y convierte cada par por mayoría en uno de 16 tipos. Produce etiquetas distintas, por ejemplo, ENTJ para ChatGPT, INTJ para GPT-4, ISTJ para BLOOM-7B e INFJ para OpenLLaMA-7B-v2, y muestra que un prompt explícito transforma ChatGPT de ENTJ a INFP, mientras BLOOM y Baichuan cambian poco. Tras continuar el entrenamiento de BLOOM y OpenLLaMA con Wikipedia china, preguntas-respuestas o APE210K, algunas cuentas y etiquetas también varían. Estos son resultados descriptivos sobre respuestas a un test y sensibilidad a intervenciones, no evidencia de que los modelos posean personalidad. La categorización es muy frágil: varias letras dependen de empates o márgenes de uno o dos ítems; el código asigna los empates sistemáticamente a I/N/F/P; GPT-4 excluye rechazos; no hay repeticiones, incertidumbre, control de orden ni validación frente a capacidad real. Los propios autores reconocen que MBTI carece de rigor, pero luego infieren razonamiento y planificación a partir de T y J sin evidencia externa. La contribución defendible es una demostración temprana de que prompts, datos, idioma y scoring alteran un perfil antropomórfico; MBTI no queda validado como evaluación de LLM.

Research question

What MBTI labels do different LLMs produce when answering a 93-item questionnaire, to what extent do those labels change with explicit instructions or examples, how do they change after continued training with three types of corpus, and can that profile reasonably serve as an indicator of model capabilities?

Method

93 binary questions associated with the four MBTI dichotomies are administered in Chinese. For open models, the logit of the last token for A and B is taken, the larger one is chosen, eight counters are accumulated, and each letter is decided by majority; for ChatGPT/GPT-4, A/B responses are requested via API, although neither version nor parameters are documented. ChatGPT, GPT-4, BLOOM-7B1, Baichuan-7B, Baichuan-13B, and OpenLLaMA-7B-v2 are compared. BLOOM and Baichuan receive an explicit role and three implicit examples; ChatGPT receives at least one role prompt. In addition, BLOOM and OpenLLaMA continue training with about 400 million tokens of Chinese Wikipedia, 400 million question-answer tokens, or 50 million of APE210K. The analysis compares counts and final types without statistical inference, counterfactual control, or an external benchmark of reasoning or planning.

Sample: Six models in the initial comparison, 93 binary questions per condition; two open models for explicit and implicit prompts, ChatGPT for explicit prompt, and two base models continued separately with three corpora. No repeated runs are reported. GPT-4 answers fewer than 93 items because its refusals are removed, with different denominators depending on the dichotomy.

Findings

  • The initial labels reported are ChatGPT-ENTJ, GPT-4-INTJ, BLOOM-ISTJ, Baichuan-7B-ENFP, Baichuan-13B-INFP, and OpenLLaMA-7B-v2-INFJ. They describe response majorities on the test, not validated psychological traits.
  • Some profiles have wide margins, such as N over S in ChatGPT (21 versus 6), but others depend on a single item or a tie: OpenLLaMA is I by 11-10; Baichuan-7B is N by 14-13; Baichuan-13B has J = P = 11 and is labeled P.
  • BLOOM changes from ISTJ to INTP under explicit prompt and also under the implicit one, but the explicit N arises from S = N = 13 and the implicit INTP includes ties S = N and J = P. The categorical change exaggerates shifts of a few items.
  • Baichuan-7B maintains ENFP under both prompts despite small count changes, which shows resistance in that configuration, not a general property of models without instruction tuning.
  • ChatGPT changes from ENTJ to INFP with the explicit role: E/I goes from 12/9 to 1/20, T/F from 15/8 to 7/18, and J/P from 12/10 to 5/17. This evidences obedience to the content of the prompt, not a transformation of an internal personality.
  • With Chinese Wikipedia, BLOOM changes ISTJ to INTP and OpenLLaMA retains INFJ; in both, S/N ends 13/13. The scoring chooses N on the tie and contributes directly to the label.
  • With question-answers, BLOOM remains INFP and OpenLLaMA INTJ; with APE210K, BLOOM ISTP and OpenLLaMA INFP. Many pairs end up tied or nearly tied, and there is no repetition to separate the effect of the corpus from training variability.
  • APE210K raises T from 13 to 15 in BLOOM and from 9 to 10 in OpenLLaMA. The article interprets this as improved reasoning, but it does not measure mathematical accuracy nor compare with a size-matched control corpus.
  • The claim that larger models favor I rests only on two uncontrolled pairs (ChatGPT/GPT-4 and Baichuan-7B/13B) that differ in training, alignment, and access; it does not isolate size.
  • The linked repository allows reconstructing the basic count and the initial results, but it does not reproduce the continued training experiments or the closed queries.

Limitations

  • MBTI has well-known problems of reliability, validity, and dichotomous typification; the authors themselves call it non-rigorous and pseudoscientific. This weakness does not disappear when applied to a language model.
  • The questionnaire forces a generative system to choose A/B on autobiographical questions. The response may reflect linguistic associations, normative preference, role-play, or instruction, not a stable disposition.
  • The 93 questions in the repository are in Chinese. Models have different Chinese competencies and amounts of pretraining, so linguistic comprehension and cultural familiarity confound the comparison of a supposed trait.
  • No validated translation, license, or official scoring manual is documented. The paper refers to a web test and the repository was published/updated after arXiv v1.
  • The ChatGPT and GPT-4 results lack model identifier, exact date of query, system prompt, temperature, seed, number of repetitions, and full responses.
  • GPT-4 refusals are selectively removed and not treated as a result. This changes the denominators per dichotomy and biases both the percentages and the comparison with models forced to answer all items.
  • The A/B order is neither randomized nor inverted, there are no parallel forms, and position or token preference is not controlled. A tendency to choose one letter can become a false MBTI preference.
  • In the code, the logit is queried for the first token of A or B encoded in isolation, not necessarily for the contextual token that would follow the prompt. This can produce invalid comparisons depending on tokenizer and space prefix.
  • The linked script prepends three generic examples to each question by default, although the article's base evaluation is presented as test administration. The effect of that context is not separated.
  • The code's `>` rule resolves all ties toward the second letter: I, N, F, and P. Several central labels, including Baichuan-13B and BLOOM conditions, depend directly on that arbitrary tiebreak.
  • Reducing each dimension to a letter discards margin and sensitivity. A change of one item or a tie can produce an apparent four-letter type transformation.
  • There is no test-retest, prompt variation, item permutation, runs with different seeds, intervals, or statistical tests. Terms such as significant are used without an inferential test.
  • The P of the J/P dichotomy means perceiving, not a p-value. The phrase attributing adaptability to an increase in P provides no statistical evidence or external criterion of adaptability.
  • The continued training experiments do not report learning rate, batch, epochs/steps, order, seeds, mixture, tokenization, or exact checkpoints; nor do they include a size- and domain-matched control corpus.
  • The corpora differ in size: Wikipedia and Q&A use about 400M tokens, APE210K about 50M. The changes cannot be cleanly attributed to data type.
  • The linked repository contains the test, the inference script, and a results JSON, but no scripts/checkpoints for the training continuations or a reproducible pipeline for ChatGPT/GPT-4.
  • The T to reasoning and J to planning inferences are circular: they are derived from the human meaning of the labels and from a few counts, without correlating with MMLU, mathematics, planning, or another independent task.
  • The claim about model size does not control for architecture, dataset, instruction tuning, RLHF, or version. Two observational comparisons do not allow inferring a scale effect.
  • Convergent/discriminant validity, internal consistency, reliability across response methods, or comparison with human judges or alternative instruments are not evaluated.
  • The model sample is small, from 2023, and centered on Chinese/English; it does not allow generalizing to current families, other languages, multimodal models, or agent environments.
  • Suggesting T/J or an INTJ type as ideal for systems or as an aid in selection may naturalize stereotypes and high-impact decisions. The article does not validate those uses nor analyze their risks.

What the study does not establish

  • It does not demonstrate that any LLM possesses personality, consciousness, internal preferences, or a stable MBTI type.
  • It does not validate MBTI as a psychometric, capability, quality, safety, or alignment metric for LLMs.
  • It does not demonstrate that T measures reasoning, that J measures planning, or that INTJ is a superior profile for an AI.
  • It does not causally demonstrate that a type of corpus produces a trait; it only observes changes after insufficiently controlled training.
  • It does not demonstrate that large models are more introverted or that instruction tuning is the sole cause of obeying a role.
  • It does not justify employment, educational, clinical, product, or governance decisions based on these labels.

Traceability

Scope: Full text

Version: arXiv:2307.16180v1 (30 Jul 2023)

Consulted source: https://arxiv.org/pdf/2307.16180

Review: Codex editorial review, 2026-07-14

Approval: Codex fidelity pass, 2026-07-14

English translation: approved, 2026-07-18

Models evaluated

  • ChatGPT (unspecified 2023 version)
  • GPT-4 (unspecified 2023 version; refusals removed)
  • BigScience BLOOM-7B1
  • Baichuan-7B
  • Baichuan-13B-Base
  • OpenLLaMA-7B-v2

Instruments and metrics

  • 93-item Chinese forced-choice MBTI questionnaire
  • Four MBTI dichotomy majority scores (E/I, S/N, T/F, J/P)
  • Last-token A/B logit comparison for open models

Data used

  • Chinese Wikipedia continuation corpus (~400M tokens)
  • Chinese question-and-answer continuation corpus (~400M tokens)
  • APE210K examination corpus (~50M tokens)
  • Linked repository MBTI question set (93 Chinese items)

Evidence and location

  • Objectives, four questions, and stated caution about MBTI: arXiv v1, pp. 1-2, abstract and introduction
  • 93-item questionnaire and A/B logits algorithm: arXiv v1, p. 3, Algorithm 1 and section 3.2
  • Models, initial types, and exclusion of GPT-4 refusals: arXiv v1, pp. 4-5, section 4.1, Figure 2, Table 3 and footnote 11
  • Results of explicit and implicit prompts: arXiv v1, pp. 5-6, section 4.2 and Tables 4-5
  • Descriptive effects of three training corpora: arXiv v1, pp. 6-7, section 4.3 and Tables 6-7
  • Approximate corpus sizes: arXiv v1, pp. 6-7, sections 4.3.1-4.3.3
  • Acknowledgment of pseudoscience and T/J inferences: arXiv v1, p. 7, section 4.4
  • Conclusions and stated limitations: arXiv v1, pp. 7-8, conclusion and limitations
  • 93 questions in Chinese and label balance: Linked repository commit 5464a2c16b73d1c8382f40fe48682830a87dfe65, mbti_questions.json
  • Systematic tiebreak toward I/N/F/P: Linked repository commit 5464a2c16b73d1c8382f40fe48682830a87dfe65, get_llms_mbti.py, get_model_examing_result
  • Non-contextual selection of the first A/B token: Linked repository commit 5464a2c16b73d1c8382f40fe48682830a87dfe65, get_llms_mbti.py, get_model_answer
  • Three generic examples prepended by default: Linked repository commit 5464a2c16b73d1c8382f40fe48682830a87dfe65, get_llms_mbti.py, few_shot_examples
  • Limited scope of the code and manual closed queries: Linked repository commit 5464a2c16b73d1c8382f40fe48682830a87dfe65, LLM/llms_mbti/readme.md
  • Initial published counts per model: Linked repository commit 5464a2c16b73d1c8382f40fe48682830a87dfe65, llms_mbti.json