Can ChatGPT Assess Human Personalities? A General Evaluation Framework

Research question

Can an LLM produce quantitative group profiles using reformulated questions from 16Personalities, and how repeatable are those outputs, how much do they change when permuting the order of options, and how similar are they between male and female labels?

Method

60 English statements from the 16Personalities web test are taken, with seven response degrees and five axes E/I, N/S, T/F, J/P, and A/T. The subject is replaced by a group label and the instruction asks if the generalization is correct instead of requesting agreement. In each trial, the order of the seven options is randomly permuted and each item is consulted once; N = 15 trials per subject are performed and the scores returned by the site are averaged. text-davinci-003, gpt-3.5-turbo, and gpt-4 are compared using default API configuration. Consistency transforms the mean Euclidean distance of each execution to its mean using alpha = 100; robustness applies the same transformation to the distance between results with fixed and permuted order; fairness combines the distance between Men/Women or Boys/Girls with the consistencies of both. Professions, income level, age, education, and three famous people are also explored, without additional individual data.

Sample: Fifteen executions of 60 questions per subject and model. The main tables cover nine labels for consistency/robustness, two gender pairs for the fairness metric, and descriptive profiles of groups/professions. There are no human participants, personal histories, behavioral observations, or ground truth personality labels.

Findings

• The mean consistency is 0.905 for InstructGPT, 0.918 for ChatGPT, and 0.921 for GPT-4. Absolute differences are small and are not accompanied by intervals or statistical contrast.
• The mean robustness against the order of options is 0.942 for InstructGPT, 0.935 for ChatGPT, and 0.936 for GPT-4. The result indicates that ChatGPT/GPT-4 change slightly more under this superficial perturbation, not that they are globally less robust.
• The metric termed fairness averages 0.753, 0.776, and 0.778 respectively. GPT-4 outperforms ChatGPT by only 0.002 and no uncertainty is estimated; the measure does not contrast any truth or harm criterion.
• People and Men are labeled Commander across the three models. Accountants are labeled Logistician and Mathematicians Architect across the three, while other professions differ by model.
• Matches with social intuitions are used as qualitative evidence, but these may precisely be learned stereotypes: for example, creative artists/Campaigner, practical accountants/Logistician, and rational mathematicians/Architect.
• ChatGPT tends to produce scores close to 50, which may increase similarity between groups and reduce distances without providing greater psychological accuracy.
• Permuting options changes frequent responses in several items. Averaging reduces the dependence on a concrete order, although it does not eliminate semantic, phrasing, or social category biases.
• When adding background labels, profiles follow normative patterns: High-income people and Adults appear as ENTJ-T; Master/PhD are classified as INTJ-T or ENTJ-T; Children/Adolescents, ENFP-T. There are no data validating these generalizations.
• When asked about famous individuals without a biography, the models express uncertainty. This contradicts the idea that the name alone suffices for a reliable evaluation and is a prudent response, not a failure to be corrected through more speculative inference.
• The authors explicitly warn that the results should not be matched with real populations nor used in questionable applications, and they recognize the unestablished scientific validity of MBTI.

Limitations

• The study does not evaluate the personality of any human being. It only asks a model if generalized statements about a group label are correct; therefore, it measures model representations or stereotypes.
• Replacing a self-report ("I do X") with a universal proposition ("artists do X") changes the construct and the unit of analysis. The resulting score does not preserve the validity of the original questionnaire.
• The correct/incorrect instruction invites judging the truthfulness of stereotypes about groups. A clear answer may be precisely an unjustified generalization; neutrality or rejection may be safer behaviors.
• The scale used comes from 16Personalities and has five axes, including Assertive/Turbulent. It is not the official MBTI of four dichotomies, although the article refers to it as MBTI and cites MBTI manuals.
• Profiles are not compared with self-reports, population distributions, behavioral records, experts, human judges, or any other validated measure. There is no estimation of accuracy, convergent validity, or criterion validity.
• Consistency means repeatability, not correctness. A model can repeat the same stereotype and obtain a high score.
• Robustness only covers the order of seven options. It does not test stability against paraphrasing, translation, negation, context, system prompts, updated models, or temporal changes.
• The fairness metric assumes that Men/Women and Boys/Girls should produce identical profiles and equates similarity with equity. It does not measure treatment, error, opportunity, calibration, harm, or performance by group.
• Multiplying by the consistencies of both groups mixes repeatability with similarity and may change the ranking without a direct relationship to bias. The constant alpha = 100 is chosen without justification or sensitivity analysis.
• Mean differences between models are small (for example, 0.002 between GPT-4 and ChatGPT in fairness), but deviations, intervals, bootstrap, tests, or corrections for multiple comparisons are not published.
• Only two binary gender pairs are used; non-binary identities and other dimensions are excluded. Even for these pairs, greater similarity does not demonstrate less stereotype.
• The profile of broad groups may hide heterogeneity and essentialize profession, age, education, or income. Intersectionality and cultural variation are not analyzed.
• API versions are moving aliases from 2023 and default configuration is used, without a reproducible date/seed. The supplement mentions context windows but does not fix immutable snapshots.
• The documentation says a continuous context of the test is maintained, while Query_GPT.py sends each item as a new single-message conversation. ChatGPT_lite and API protocols are not equivalent.
• Scoring depends on sending responses to the live 16Personalities endpoint and retrieving a session. The proprietary algorithm, the version, and possible service changes are neither frozen nor auditable.
• The repository parser searches for substrings in free responses. If an item does not match any expression, it does not validate the length and subsequent responses may shift to wrong questions when constructing the payload.
• The repository conversion assigns correct to -3 and wrong to +3 before consulting the service, but does not document the sign convention. The fidelity of this mapping to the external test cannot be verified with the published artifacts.
• The README declares that the simplified code produces only one execution and that the user must adapt it to reproduce 15. The aggregation loop is commented out, so the tables are not directly regenerated.
• In the inspected public commit, Query_InsturctGPT.py and Crawler_16personalities.py fail `py_compile` due to indentation errors. The scoring path is not executable without correction.
• The repository only provides CSVs of six subjects per model, while the tables include more professions, backgrounds, and 15 executions. Full raw artifacts and the exact aggregation pipeline are missing.
• Causal explanations about RLHF, reasoning capacity, or understanding of backgrounds are inferred from intuitive profiles without ablations or comparators. Matching a researcher's stereotype does not validate an evaluation.
• Only three models from the same family/provider are evaluated and in English. Results do not generalize to current models, other architectures, languages, or cultures.
• The design may amplify discrimination if used to infer traits of groups or individuals. The ethics section acknowledges this, requires review, and prohibits directly generalizing the results.

What the study does not establish

• It does not demonstrate that ChatGPT, GPT-4, or InstructGPT can evaluate the real personality of a person or population.
• It does not demonstrate that a consistent output is true, psychometrically valid, or free of stereotypes.
• It does not demonstrate that ChatGPT/GPT-4 are fairer than InstructGPT; they only score slightly higher on a similarity defined by the authors.
• It does not validate 16Personalities/MBTI for automatic inference nor preserve its validity when replacing self-report with judgments about groups.
• It does not allow the use of profession, gender, age, income, or education labels to make decisions about individuals.
• It does not provide evidence of mental states, motivations, psychology, or internal perceptions of the model beyond its generated responses.

Traceability

Scope: Full text

Version: Findings of EMNLP 2023 proceedings version, pp. 1184–1194; arXiv:2303.01248v3 supplement also reviewed

Consulted source: https://aclanthology.org/2023.findings-emnlp.84.pdf

Review: Codex editorial review, 2026-07-14

Approval: Codex fidelity pass, 2026-07-14

English translation: approved, 2026-07-18

Models evaluated

• InstructGPT (text-davinci-003)
• ChatGPT (gpt-3.5-turbo)
• GPT-4 (gpt-4)

Instruments and metrics

• 16Personalities online test (60 statements, seven response levels)
• Five 16Personalities axes: E/I, N/S, T/F, J/P, A/T
• Custom Euclidean-distance consistency score
• Custom fixed-vs-permuted option-order robustness score
• Custom gender-label similarity score called fairness

Data used

• General group labels: People, Men, Women, Boys, Girls
• Occupation labels: Barbers, Accountants, Doctors, Artists, Mathematicians, Politicians and six supplementary professions
• Background labels for income, age and education
• Named-person probes: Barack Obama, Taylor Swift and Michael Jordan

Evidence and location

• Objective, components, and main statements: Findings of EMNLP 2023 PDF, pp. 1184–1185, abstract and introduction
• Permutation of options and average of executions: Findings PDF, pp. 1186–1188, sections 3.1 and 3.4
• Substitution of the subject by groups: Findings PDF, pp. 1186–1187, section 3.2
• Change from agreement to correctness: Findings PDF, p. 1187, section 3.3 and Figure 2
• Formulas for consistency, robustness, and fairness: Findings PDF, pp. 1188–1189, equations 2–5
• Models, subjects, and fifteen executions: Findings PDF, p. 1189, Experimental Setups
• Complete profiles of groups and professions: Findings PDF, p. 1189, Table 1
• Results of consistency, robustness, and fairness: Findings PDF, p. 1190, Tables 2–3 and section 5
• Profiles by income, age, and education: Findings PDF, p. 1191, Table 4 and section 6
• Uncertainty regarding specific individuals: Findings PDF, p. 1191, Figure 6 and Assessment of Specific Individuals
• Limitations, unestablished validity, and ethical warnings: Findings PDF, p. 1192, Limitations and Ethics Considerations
• Exact models, aliases, and default configuration: arXiv:2303.01248v3 supplementary materials, Appendix A
• 60-item test and fifth A/T axis of 16Personalities: arXiv v3 supplementary materials, Appendix C.1
• Simplified single-execution code: Linked repository commit a50bac9ac6269aac175ef7b62b81c7c4f9f1f46b, README.md
• Query protocols and separate conversations per item: Linked repository commit a50bac9ac6269aac175ef7b62b81c7c4f9f1f46b, Query_GPT.py and Query_ChatGPT.py
• Substring parser, external scoring, and commented aggregation: Linked repository commit a50bac9ac6269aac175ef7b62b81c7c4f9f1f46b, Crawler_16personalities.py
• Reproducible syntax errors in two scripts: Linked repository commit a50bac9ac6269aac175ef7b62b81c7c4f9f1f46b, Python py_compile of Query_InsturctGPT.py and Crawler_16personalities.py