Exploring the Potential of Large Language Models to Simulate Personality

Research question

To what extent can four commercial LLMs obey instructions for high, medium, or low Big Five trait levels consistently in both a questionnaire and free text, and what human, automatic, and lexical evaluation allows measuring this?

Method

First, a high or low level of each trait is induced separately and each model responds only to the corresponding BFI-44 items; scores are aggregated and Cronbach's alpha, Guttman's lambda, and distributions by model and trait are reported. Then, for six open questions, texts are generated for each combination of four models, four temperatures (0, 0.5, 0.7, and 0.9), five traits, and five discrete scores. A subset of about 288 responses is distributed among eight annotators, three per text, who score from -2 to +2, choose the type of evidence, and mark relevant expressions; agreement is calculated with Fleiss' kappa and final labels by voting. GPT-4o automatically classifies the entire corpus with a prompt inspired by CARP and is compared with human labels at three levels. Detectable responses are summarized in confusion matrices; TF-IDF, nearest neighbors, and spaCy lemmatization are used to analyze lexical separation. The audit contrasted the figures, figures, prompts, and claims of the PDF with the public repository and with the official records of arXiv, CustomNLP4U, and ACL Anthology.

Sample: Four LLMs are compared. In free text, the described design crosses 4 models by 4 temperatures by 5 traits by 5 scores by 6 questions, which implies 2,400 expected responses; Figure 4 counts 2,399 classified or nondistinguishable and leaves one Claude-Openness sample unexplained. Approximately 288 texts, 12% of the expected total, were annotated by eight people and each text received three ratings. The article says it balances that subset by model and temperature, but does not document balance by trait, score, or question. For multi-trait profiles, means and variances are adjusted to 1,015,342 human responses from the Big Five Personality Test Dataset, although that second set is reserved for future work and does not enter the main results.

Findings

• The four models can partially change their BFI-44 responses when ordered to represent a high or low trait.
• Figure 1 shows frequent overlap between high and low conditions, such that no model consistently separates the five traits.
• Claude better separates Openness and Conscientiousness in the questionnaire according to the authors' interpretation.
• Mixtral shows the clearest separation for Agreeableness and Neuroticism in the questionnaire according to the authors.
• The two GPTs better distinguish the extremes of Extraversion; GPT-4o also better separates Agreeableness.
• The reported Cronbach's alpha ranges from 0.87 to 0.94 across traits.
• The reported Guttman's lambda ranges from 0.95 to 0.97.
• These coefficients indicate item consistency under the protocol, not that the responses are real or psychologically valid.
• No validity measure is published even though the method claims to measure reliability and validity.
• Text generation uses six open questions, five scores per trait, and four temperatures.
• The described factorial design implies 2,400 single-trait texts.
• Figure 4 sums 2,399 cases, with 119 in Claude-Openness and 120 in each of the other 19 model-trait combinations.
• The single missing observation is not explained.
• Human agreement at Level 1 ranges between kappa 0.59 for Neuroticism and 0.71 for Agreeableness.
• Human agreement at Level 2 ranges between 0.57 for Neuroticism and 0.70 for Agreeableness.
• These values reflect moderate agreement, not an unequivocal psychological label.
• The GPT-4o classifier obtains weighted F1 of Level 1: 0.87 in Openness, 0.80 in Conscientiousness, 0.76 in Extraversion, 0.71 in Agreeableness, and 0.63 in Neuroticism.
• At Level 2, F1 drops to 0.78, 0.75, 0.65, 0.67, and 0.50 respectively.
• The MAE of Level 3 is 0.44 in Openness, 1.05 in Conscientiousness, 0.88 in Extraversion, 0.54 in Agreeableness, and 1.77 in Neuroticism.
• Neuroticism is simultaneously the trait with the worst human agreement and the worst classifier performance.
• Automatic evaluations of Neuroticism are therefore especially uncertain.
• Figure 4 marks Openness as detectable in 116 of 119 Claude texts, 117 of 120 GPT-3.5, 107 of 120 GPT-4o, and 107 of 120 Mixtral.
• For Neuroticism, detectability drops to 61 of 120 in Claude, 74 in GPT-3.5, 79 in GPT-4o, and 72 in Mixtral.
• Agreeableness is detectable in 110 of 120 Claude texts, 107 GPT-3.5, 107 GPT-4o, and 108 Mixtral.
• High detectability of a trait does not imply that the requested level is classified correctly.
• The matrices in Figure 2 remove the texts that the classifier calls Nondistinguishable.
• That exclusion conditions the apparent accuracy to the cases in which the automatic evaluator already found a signal.
• Medium levels are frequently confused with high or low extremes.
• The authors conclude that a binary coding may be more controllable than five fine levels.
• Openness is the most consistently expressed trait in the corpus according to human and automatic evaluation.
• Neuroticism is the most difficult and presents a general bias toward low scores.
• Several models show bias toward high Agreeableness even when the opposite extreme is requested.
• GPT-3.5 shows biases toward high Conscientiousness and Agreeableness and toward high Extraversion in part of the low conditions.
• GPT-4o better controls Openness, Conscientiousness, and Extraversion, but maintains bias toward high Agreeableness.
• Mixtral detects Conscientiousness in only 68 of 120 texts and Extraversion in 63 of 120.
• Claude does not always follow the format: sometimes it recommends instead of answering or explicitly reveals the trait.
• Those Claude responses were manually edited before analysis.
• It is not reported how many responses were edited, nor are original and modified versions preserved.
• Figure 3 suggests that Claude and GPT-4o better lexically separate the extremes of score.
• The lexical analysis reports that 16% of the patterns identified in the texts comes from the prompts.
• That overlap quantifies direct leakage of the trait definition into the evaluated signal.
• Nouns and adjectives differentiate levels better than verbs in the presented analysis.
• Except for Neuroticism, neutral texts share up to 34% of patterns with low levels according to the manuscript.
• The same trait descriptions feed generation and the GPT-4o classifier, creating semantic circularity.
• The article acknowledges that answering questionnaires and generating text may produce incompatible profiles.
• The proposed explanation is that the default assistant role favors high Agreeableness and low Neuroticism.
• The public repository contains a notebook and modules for BFI-44, generation, classification, and visualization.
• The repository does not contain the generated texts, human labels, classifier outputs, or the source figures promised as a dataset.
• The public notebook has no saved outputs and configures only temperature 0.7 for text, not the four temperatures of the article.
• The repository allows repeating the general idea, but does not reproduce the published results without additional data, versions, dependencies, and parameters.

Limitations

• The full-text version is arXiv v1 and no later revision exists.
• The work appears as accepted under the archival heading of CustomNLP4U 2024.
• It has no individual record, pages, or DOI in the ACL Anthology proceedings.
• The absence of the individual record is not explained on the workshop site or in the preprint.
• The README claims presentation at the workshop, but still says it will link the article after publication.
• The year 2025 corresponds to the preprint, while the declared acceptance and presentation are from 2024.
• The Big Five is incorrectly described as five dichotomies although its traits are continuous.
• The text claims that recent research adds Honesty-Humility to the Big Five.
• Honesty-Humility usually belongs to the HEXACO model and the citation of Costa and McCrae 2008 does not support that addition.
• The claim of cross-cultural validity of the Big Five relies on a 1985 reference on the logical validity of the MBTI, not on a cross-cultural test of the BFI.
• The operational definitions of trait used in the code come from a popular science page of Simply Psychology.
• No facets or descriptors validated specifically for text generation are used.
• The BFI-44 was designed for human self-report, not to measure entities without subjective experience.
• The questionnaire explicitly asks to act as a person with the high or low trait.
• Then it evaluates only the items of the same trait revealed in the prompt.
• That task measures obedience and semantic knowledge of the questionnaire more than latent personality.
• There is no promptless condition for each model in the main analysis.
• There is no comparison with human participants receiving equivalent instructions.
• Test-retest stability of the same configuration is not evaluated.
• The number of replicates per item and questionnaire condition is not reported.
• It is not reported how alpha and lambda were calculated from model observations.
• Confidence intervals for alpha or lambda are not reported.
• Alpha may increase due to item redundancy and does not demonstrate a real construct.
• Lambda also does not demonstrate criterion, convergent, or predictive validity.
• The phrase real and plausible does not follow from internal consistency coefficients.
• No test of the validity that the method announces is reported.
• Figure 1 lacks confidence intervals and visible sample sizes.
• The high-low separation of Figure 1 is not formally quantified.
• Claims of accurate performance are based on undefined visual inspection.
• Commercial models are not identified by snapshot except Claude.
• The aliases GPT-3.5 Turbo and GPT-4o may change over time.
• The exact provider or version of Mixtral is not specified in the article; the code uses an alias of the Mistral API.
• Exact dates of calls to each model are not reported.
• Parameters other than temperature are not reported.
• Top-p, seed, penalties, or comparable maximum length are not reported.
• Differences in security policy between providers are not controlled.
• Differences in length between responses are not controlled.
• The text design expects 2,400 outputs, but Figure 4 contains 2,399.
• The missing Claude-Openness output is not documented.
• API errors, rejections, or parsing failures are not explained.
• The public function literally saves GPT Fail when retries are exhausted.
• It is not shown whether those failures were filtered in the study.
• Claude received manual editing when it revealed traits or did not respond directly.
• No prior protocol for that editing is defined.
• It is not reported how many cases were modified.
• It is not evaluated how much the metrics change before and after editing.
• Manual editing is applied to one model and may break comparability.
• Raw and edited texts are not publicly preserved.
• The multi-trait set is mentioned but is reserved entirely for future work.
• The public code only implements generation of one trait at a time.
• The number and license of the multi-trait dataset are not reported.
• The means and variances of the human dataset are not published in the article.
• Adjusting normal distributions per trait does not necessarily preserve correlations between traits.
• It is not explained whether the human scores were cleaned or weighted.
• The Big Five Personality Test Dataset is a voluntary online sample, not representative.
• A human partition is not used as a reference for language or behavior.
• Only about 288 of 2,400 expected responses are annotated.
• About 288 leaves the exact size ambiguous.
• The subset is balanced by model and temperature, but is not documented by trait, score, or question.
• The sampling method for the 288 texts is not described.
• It is not reported whether annotators were blind to the model.
• It is not reported whether they were blind to the requested score.
• It is not reported whether they saw the original question alongside the response.
• The complete annotation interface is not published.
• The 864 expected individual ratings are not published.
• The final votes per text are not published.
• The marked linguistic fragments are not published.
• The training of the eight annotators is not reported.
• Experience in psychology or linguistics is not reported.
• Age, gender, language, or culture of the annotators is not reported.
• Compensation is not reported.
• Consent is not reported.
• Ethical approval or exemption is not reported.
• Three raters per text allow voting, but a majority of two can mask high uncertainty.
• Fleiss' kappa between 0.57 and 0.71 does not indicate perfect agreement.
• Confidence intervals for kappa are not reported.
• Systematic bias per annotator is not analyzed.
• Reliability of the underlined expressions is not calculated.
• GPT-4o is selected as classifier for superior performance without showing the selection comparison.
• The classifier is evaluated on the same human subset used to justify its extrapolation.
• There is no independent human validation set.
• It is not reported whether the classifier prompt was adjusted after seeing those labels.
• GPT-4o also classifies texts generated by GPT-4o, creating family self-evaluation.
• Generator and classifier receive the same trait definitions.
• The 16% of patterns inherited from the prompt evidences feature leakage.
• The classifier may reward semantic repetition of the prompt rather than coherent personality.
• Weighted F1 hides minority class performance.
• Per-class supports are not published in Table 2.
• Human-classifier confusion matrices for the three levels are not published.
• Overall accuracy or macro-F1 is not reported.
• The F1 of 0.50 and MAE 1.77 for Neuroticism limit its use as an automatic label.
• The worst classifier performance coincides with the main substantive finding on Neuroticism.
• Classifier uncertainty is not propagated to conclusions per model.
• Nondistinguishable cases are excluded from Figure 2.
• Excluding abstentions may inflate the apparent precision of detected levels.
• Figure 4 shows detectability separately, but does not offer a joint metric of coverage and accuracy.
• The classifier is not compared with a simple lexical rule or a supervised classifier.
• It is not compared with an additional human evaluator on the entire corpus.
• No statistical tests are performed between models.
• No statistical tests are performed between temperatures.
• Confidence intervals for detection proportions are not reported.
• No correction for multiple comparisons is applied.
• Sensitivity to other prompt wordings is not reported.
• Sensitivity to other trait definitions is not reported.
• Sensitivity to other six questions is not reported.
• The six questions cover preferences and aspirations, not prolonged conversation.
• Consistency between turns is not evaluated.
• Personality memory is not evaluated.
• Resistance to conflicting instructions is not evaluated.
• Interaction between traits is not evaluated.
• Complex profiles with all five traits are not evaluated in the results.
• Languages other than English are not evaluated.
• Cultures other than the studied one are not evaluated.
• Tasks with behavioral consequences are not evaluated.
• Empathy or user satisfaction is not evaluated despite motivating them in the introduction.
• Safety of inducing high Neuroticism or other sensitive traits is not evaluated.
• Stigmatization of high or low levels is not studied.
• It is not validated that the responses appear written by humans with known BFI scores.
• A human corpus with personality ground truth for comparison is not included.
• TF-IDF similarity may reflect the shared question more than the trait.
• The code searches for global neighbors and then filters those sharing the trait, an operation distinct from searching for five neighbors within each trait.
• The public code visualizes mean similarity, while the article describes mean scores of the five most similar texts.
• The exact code that generated Figure 3 is not published.
• SVD of 500 components is not justified or analyzed for sensitivity.
• TF-IDF is not compared with semantic embeddings.
• The lemma analysis does not control for base frequency or length.
• No statistical test for differences between nouns, adjectives, and verbs is reported.
• The 34% overlap is not accompanied by a denominator or interval.
• Appendix D presents word lists without frequencies or complete selection criteria.
• Some visible lemmas are truncated or are artifacts, such as bogge and extroverte.
• The repository was last updated before the preprint and contains no subsequent changes that close reproduction.
• The repository does not contain the dataset that the PDF says is available at the same link.
• The repository does not contain questionnaire results.
• The repository does not contain generated texts.
• The repository does not contain human annotations.
• The repository does not contain classifier outputs.
• The notebook does not contain executed outputs.
• There is no requirements.txt, pyproject.toml, or lockfile.
• Dependencies are not versioned.
• The notebook installs only three SDKs, although the code requires pandas, NumPy, matplotlib, seaborn, and scikit-learn.
• The fallback path for invalid BFI responses uses a variable pipe that is not defined in the repository.
• That branch produces a NameError if a model does not return exactly one allowed option.
• The classifier parser attempts to repair JSON with ad hoc substitutions without validating types or ranges.
• There are no automated tests.
• There is no continuous integration.
• There are no example data with expected results.
• The public notebook configures only temperature 0.7 for text generation.
• The notebook configures only temperature 0.9 for the questionnaire.
• The article uses four temperatures and the repository does not document how to obtain exactly that execution.
• There are no reproducible NumPy or sampling seeds.
• Response identifiers from the APIs are not recorded.
• Hashes of prompts or configuration are not recorded.
• Cost, tokens, latency, or execution dates are not reported.
• Dependence on commercial APIs prevents reproducing historical snapshots without the original outputs.
• There is no preregistration.
• There is no independent replication.
• The limitations declared by the authors omit most of these measurement and reproduction problems.

What the study does not establish

• It does not demonstrate that LLMs have psychological personality.
• It does not demonstrate that an LLM's BFI-44 responses are real or plausible for exceeding alpha 0.85.
• It does not demonstrate construct, convergent, discriminant, or predictive validity.
• It does not demonstrate trait stability across sessions.
• It does not demonstrate stability between questionnaire and free generation.
• It does not demonstrate complete profiles of five traits because the main analysis induces one at a time.
• It does not demonstrate fine control of five levels, since medium levels are confused with extremes.
• It does not demonstrate that Openness or Neuroticism behave the same under other prompts or languages.
• It does not demonstrate that low Neuroticism reflects personality rather than assistant policy and safety.
• It does not demonstrate that high Agreeableness is a stable trait rather than conversational alignment.
• It does not demonstrate that the GPT-4o classifier is an independent measure of the generator.
• It does not demonstrate classifier accuracy for Neuroticism.
• It does not demonstrate that excluding nondistinguishable cases preserves an unbiased comparison.
• It does not demonstrate that manual editing of Claude does not alter the results.
• It does not demonstrate that TF-IDF similarity equates to personality similarity.
• It does not demonstrate that lexical patterns do not come from the prompt; the study itself attributes 16% to that source.
• It does not demonstrate that generated texts resemble texts from people with known BFI.
• It does not demonstrate human behavior beyond six brief questions.
• It does not demonstrate dialogue quality, empathy, engagement, or user satisfaction.
• It does not demonstrate utility in multi-agent systems, games, or business applications.
• It does not demonstrate safety for psychological personalization.
• It does not demonstrate cultural or multilingual generalization.
• It does not demonstrate reproducibility of figures and figures with the published repository.
• It does not demonstrate availability of the dataset that the article claims to release.
• It does not establish an individual publication in ACL Anthology or a DOI for the article.
• It does not justify using these outputs as substitutes for human psychological evaluations.

Traceability

Scope: Full text

Version: arXiv 2502.08265v1, 12 Feb 2025; open 12-page preprint after CustomNLP4U 2024 acceptance

Consulted source: https://arxiv.org/pdf/2502.08265v1

Review: Codex full-text, bilingual-fidelity, visual, bibliographic, workshop-proceedings, psychometric-validity, human-annotation, metric, code-artifact, reproducibility, ethics and internal-consistency audit, 2026-07-15

Approval: Codex fidelity pass, 2026-07-15

English translation: approved, 2026-07-18

Models evaluated

• GPT-3.5 Turbo
• GPT-4o
• Claude 3 Haiku (claude-3-haiku-20240307 in the repository)
• Mixtral 8x22B (open-mixtral-8x22b in the repository)

Instruments and metrics

• Big Five Inventory 44 (BFI-44)
• Cronbach's alpha
• Guttman's lambda
• Fleiss' kappa
• Weighted precision, recall and F1
• Mean absolute error (MAE)
• GPT-4o CARP-style trait classifier
• TF-IDF nearest-text analysis
• spaCy lemmatization and part-of-speech analysis

Data used

• Big Five Personality Test Dataset (1,015,342 questionnaire responses)
• Six open-ended personality questions
• LLM-generated single-trait text corpus
• Human annotation subset of about 288 generated texts
• Generated multi-trait profiles sampled from fitted normal distributions, mentioned but not analyzed

Evidence and location

• Identity and citable version: arXiv 2502.08265v1, submitted 12 Feb 2025; title page lists Maria Molchanova, Anna Mikhailova, Anna Korzanova, Lidiia Ostyakova and Alexandra Dolidze
• Full text inspected: .cache/editorial-sources/article-088/source.pdf; 12 pages; sha256 0160d1f7d12b4e699d07bd7efa474a68551fd985fad2e30d2c01c9aa3459c3f5
• Workshop acceptance: Official CustomNLP4U 2024 Accepted Papers page, archival section, lists the title and five authors; workshop held at EMNLP on 16 Nov 2024
• Absence of individual record in proceedings: ACL Anthology CustomNLP4U 2024 volume and event index inspected 15 Jul 2026: no title or Molchanova record; no paper DOI or pages found
• Question and general design: Abstract, Introduction and Section 3, pp. 1-2
• Induced questionnaire: Section 4 and Appendix C.1, pp. 2-3 and 10; high/low single-trait prompts over corresponding BFI-44 items
• Reported reliability: Section 4 p. 3 and Table 4 p. 10: alpha 0.87-0.94, lambda 0.95-0.97; prose calls responses real and plausible
• Questionnaire separation: Figure 1 and adjacent prose, p. 3
• Text generation design: Section 5.1 and Appendix C.2, pp. 3-4 and 10-11: six questions, scores 1-5 and temperatures 0, 0.5, 0.7, 0.9
• Manual editing of Claude: Section 5.1, p. 4: direct trait disclosures were masked and task-misaligned content removed
• Sample and human annotation: Section 5.2, p. 4: about 288 texts, eight annotators, three raters per text, -2 to +2 scores, voting and highlighted patterns
• Human agreement: Table 1, p. 4: Fleiss kappa Level 1 0.59-0.71 and Level 2 0.57-0.70
• Automatic classifier: Section 5.3 and Appendix C.3, pp. 4-5 and 11: GPT-4o CARP-style clues, reasoning, score and decision type
• Classifier performance: Tables 2-3, p. 5: weighted F1 Level 1 0.87-0.63; Level 2 0.78-0.50; MAE 0.44-1.77
• Matrices conditioned on detectability: Figure 2, pp. 5-6, and public code experiment_functions/qa_text_generation.py: rows with decision type Nondistinguishable are removed before crosstab
• Detectability counts: Figure 4, p. 10: 2,399 total cases; Claude-Openness 3 nondistinguishable plus 116 detected; all other model-trait panels total 120
• Findings by model and trait: Section 5.3, pp. 5-7; Figure 2 confusion matrices and Figure 5 nondistinguishable breakdown
• TF-IDF analysis: Section 5.4 and Figure 3, pp. 6-7
• Lexical leakage from prompts: Section 5.4, p. 7: 16% of linguistic patterns derived from prompts and up to 34% shared by neutral and low lexicons
• Limited conclusion of the study: Discussion and Conclusion, p. 8: task-dependent behavior, Agreeableness/Neuroticism bias and questionnaire-text inconsistency
• Declared limitations: Limitations, p. 8: Big Five assumptions, unmodeled size/training-data effects and partial human annotation
• Public repository: https://github.com/mary-silence/simulating_personality at commit 9c02fd5c757928f66cdb338bac0c3d22fcd22e10, last push 12 Nov 2024; inspected 15 Jul 2026
• Dataset and results absent: Repository tree at commit 9c02fd5: README, one notebook, source modules, BFI-44 items, trait definitions and six questions; no results, generated dataset or annotations
• Non-equivalent configuration: main.ipynb at commit 9c02fd5: text generation temperatures [0.7], questionnaire temperature 0.9 and no saved outputs; paper reports four text temperatures
• Reproducible code defect: experiment_functions/questionnaire.py match_score references undefined pipe on invalid model answers; repository contains no pipe definition
• Dependencies and QA absent: Repository commit 9c02fd5 has no requirements file, lockfile, tests or CI; notebook installs only openai, anthropic and mistralai
• Similarity analysis discrepancy: Public experiment_functions/qa_text_generation.py computes global top-5 neighbors then filters same-trait and plots mean similarity matrices; paper Figure 3 describes averaged trait scores of five most similar texts
• Integral reading and visual verification: All 12 pages rendered and inspected, including Figures 1-5, Tables 1-4, prompts, Appendix D, limitations and references; checked 15 Jul 2026