Synthetic Personality Review

Developing and Improving Personality Inventories Using Generative Artificial Intelligence: The Psychometric Properties of a Short HEXACO Scale Developed Using ChatGPT 4.0

Evaluation and psychometric validity2025Taylor & FrancisApproved editorial review

Authors: Ard J. Barends, Reinout E. de Vries

Keywords: Artificial Intelligence, Generative AI, HEXACO personality inventory, ChatGPT 4.0, Psychometrics, Survey development, Content validity

Source: Open primary source (opens in a new tab)

2
Authors
26
Findings
73
Limitations
15
Evidence

Editorial summary

English

This exploratory study evaluates whether ChatGPT 4.0 can help create and then improve a brief personality questionnaire. The authors supplied definitions of the six HEXACO domains and 24 facets plus item-writing rules, generated a 24-item ChatGPT HEXACO Inventory, and produced three versions: a baseline (CHI-B), one instructed to increase internal consistency (CHI-R), and one instructed to increase content validity (CHI-V). Items were generated in English, corrected through new instructions when the authors detected two problems, translated into Dutch by ChatGPT, and corrected again after eight translation problems. The workflow therefore includes substantive human oversight and is not autonomous generation of a validated scale.

All 682 participants completed the HEXACO-60 and the human-developed Brief HEXACO Inventory and were randomly assigned to CHI-B (n=228), CHI-R (n=242), or CHI-V (n=212). Within the CHI-B group, mean alpha was .51 versus .53 for the BHI, with no difference, and mean convergent correlation with the HEXACO-60 was .68 versus .72, also with no difference. The FDR-corrected exception was Honesty-Humility: CHI-B correlated .50 with the HEXACO-60 versus .65 for BHI (z=2.90, p=.026). Mean discriminant correlations and associations with authoritarianism and social dominance orientation did not differ significantly either. These results make CHI-B similar to BHI on the tests used, but do not establish equivalence or superiority.

The improvement instructions did not produce the intended effect. Mean alphas were .56 for CHI-R and .50 for CHI-V versus .51 for CHI-B; neither mean difference was significant. Mean convergent correlations were .66 and .63 versus .68, and discriminant correlations did not improve. CHI-V actually reduced the internal consistency of Emotionality and Conscientiousness significantly relative to CHI-B; its Emotionality alpha was .22. In comparisons with BHI, only 2 of 35 CHI-R contrasts and 3 of 35 CHI-V contrasts were significant after FDR correction; all three CHI-V results concerned Emotionality and were unfavorable or divergent relative to BHI.

The proper interpretation is narrow. The scales have only four items per domain and several low alphas; authoritarianism reaches only .48. There is no factor analysis, IRT, measurement invariance, test-retest assessment, cognitive interviewing, formal expert content review, or equivalence/noninferiority design. “Content validity” is approached mainly through convergence, discrimination, and two external criteria rather than a comprehensive content-validity procedure. Data and code are not public, the exact ChatGPT 4.0 snapshot is unidentified, and one generation per version cannot represent system variability. Two negative-keying errors survived final oversight: Honesty-Humility in all three versions and Agreeableness in CHI-V had no reverse-keyed items.

The defensible contribution is that a ChatGPT-plus-human-review workflow produced a short scale with some properties close to the BHI in this Dutch sample, while generic instructions to “improve” reliability or content offered no psychometric shortcut. The paper does not study ChatGPT's personality; it studies ChatGPT as an item-writing tool for measuring human personality. The official prompt-and-item supplement was located at Taylor & Francis, but the publisher returned 403 when the file was frozen outside the page and Chrome was not open; transparently, this audit makes no claim that depends on that DOCX. The conclusions reported here derive from the complete publisher PDF, its tables, and its notes.

Español

Este estudio exploratorio evalúa si ChatGPT 4.0 puede ayudar a crear y después mejorar un cuestionario breve de personalidad. Los autores proporcionaron definiciones de los seis dominios y 24 facetas HEXACO y reglas de redacción, generaron un ChatGPT HEXACO Inventory de 24 ítems y obtuvieron tres versiones: una base (CHI-B), otra instruida para aumentar consistencia interna (CHI-R) y otra para aumentar validez de contenido (CHI-V). Los ítems se generaron en inglés, se corrigieron mediante nuevas instrucciones cuando los autores detectaron dos problemas, se tradujeron al neerlandés con el propio ChatGPT y se volvieron a corregir ante ocho problemas de traducción. Por tanto, el proceso incluye supervisión humana sustantiva y no es generación autónoma de una escala validada.

Los 682 participantes completaron HEXACO-60 y el Brief HEXACO Inventory humano, y fueron asignados aleatoriamente a CHI-B (n=228), CHI-R (n=242) o CHI-V (n=212). En el grupo CHI-B, el alfa medio fue 0,51 frente a 0,53 en BHI, sin diferencia, y la correlación convergente media con HEXACO-60 fue 0,68 frente a 0,72, también sin diferencia. La excepción corregida por FDR fue Honesty-Humility: CHI-B correlacionó 0,50 con HEXACO-60 frente a 0,65 para BHI (z=2,90; p=0,026). Las correlaciones discriminantes medias y las asociaciones con autoritarismo y orientación a la dominancia social tampoco difirieron significativamente. Estos resultados hacen al CHI-B parecido al BHI en las pruebas usadas, pero no demuestran equivalencia ni superioridad.

Las instrucciones de mejora no produjeron el efecto buscado. Los alfa medios fueron 0,56 para CHI-R y 0,50 para CHI-V frente a 0,51 en CHI-B; ninguna diferencia media fue significativa. Las correlaciones convergentes medias fueron 0,66 y 0,63 frente a 0,68, y tampoco mejoraron las correlaciones discriminantes. CHI-V incluso redujo significativamente la consistencia de Emotionality y Conscientiousness frente a CHI-B; su alfa de Emotionality fue 0,22. Al comparar las versiones modificadas con BHI, solo 2 de 35 contrastes para CHI-R y 3 de 35 para CHI-V fueron significativos tras FDR; los tres de CHI-V se concentraron en Emotionality y fueron desfavorables o divergentes respecto a BHI.

La lectura correcta es limitada. Las escalas tienen solo cuatro ítems por dominio y varios alfa bajos; autoritarismo alcanza apenas 0,48. No hay análisis factorial, IRT, invariancia, test-retest, entrevistas cognitivas, evaluación experta formal de contenido ni estudio de equivalencia o no inferioridad. “Validez de contenido” se aproxima principalmente mediante convergencia, discriminación y dos criterios externos, no mediante un procedimiento integral de validez de contenido. Tampoco se publican datos ni código, no se identifica el snapshot exacto de ChatGPT 4.0 y una sola generación por versión no representa la variabilidad del sistema. Dos errores de clave negativa pasaron la supervisión final: Honesty-Humility en las tres versiones y Agreeableness en CHI-V quedaron sin ítems invertidos.

La aportación defendible es que un flujo ChatGPT más revisión humana produjo una escala breve con algunas propiedades próximas a BHI en esta muestra neerlandesa, mientras instrucciones generales de “mejorar” fiabilidad o contenido no ofrecieron un atajo psicométrico. El trabajo no estudia la personalidad de ChatGPT, sino su uso como herramienta de redacción de ítems para medir personalidad humana. El suplemento oficial con prompts e ítems fue localizado en Taylor & Francis, pero el editor devolvió 403 al intentar congelarlo fuera de la página y Chrome no estaba abierto; por transparencia, la auditoría no atribuye ningún detalle que dependa de ese DOCX. Las conclusiones aquí recogidas proceden del PDF editorial completo, sus tablas y notas.

Research question

Can ChatGPT 4.0 generate a brief HEXACO inventory with psychometric properties comparable to those of the human BHI and improve its internal consistency or content validity in a targeted manner when requested?

Method

Exploratory validation study of three 24-item CHI inventories in Dutch. All participants completed HEXACO-60 and BHI and were randomized to CHI-B, CHI-R, or CHI-V; they then completed authoritarianism and SDO. Alpha, convergent correlations with HEXACO-60, discriminant correlations, and relations with two criteria were compared, with False Discovery Rate correction. The audit read and rendered the nine pages of the file, repository and publisher covers plus pages 419-425, checked Tables 1-2, notes, discussion, data availability, and bibliography, and verified the full publisher page. The official supplementary DOCX was identified by name and URL, but could not be frozen due to the publisher's 403 control and is not used as evidence of additional details.

Sample: 693 people were recruited through the personal networks of undergraduate students. Eleven were excluded for non-conforming patterns on HEXACO-60 or for taking on average less than one second per item, leaving N=682: 223 men, 457 women, and 2 people of another category, with a mean age of 35.28 years and SD=17.66. All completed HEXACO-60 and BHI; random assignment distributed 228 to CHI-B, 242 to CHI-R, and 212 to CHI-V. The analytical unit is a person, not a model response: ChatGPT only generated the items.

Findings

  • The article appeared online on 27 December 2024 and was published in Journal of Personality Assessment 107(4), pages 419-425, with a CC BY 4.0 license.
  • The final sample was 682 participants after excluding 11 of 693 responses for compliance and speed controls.
  • Randomization assigned 228 participants to CHI-B, 242 to CHI-R, and 212 to CHI-V.
  • ChatGPT initially generated 24 items, four per HEXACO domain and one per facet.
  • The authors detected two problems in the English generation and eight in the Dutch translation and requested new versions.
  • Domain reliabilities for CHI-B ranged between 0.31 and 0.63.
  • The mean alpha of CHI-B was 0.51 and that of BHI 0.53; chi-square(1)=0.09 and p=0.902.
  • The mean convergent correlation with HEXACO-60 was 0.68 for CHI-B and 0.72 for BHI; z=0.93 and p=0.492.
  • Honesty-Humility had lower convergence in CHI-B, r=0.50, than in BHI, r=0.65; z=2.90 and p=0.026 after FDR.
  • The mean absolute discriminant correlation was 0.12 for CHI-B and 0.09 for BHI, with no difference, z=0.43 and p=0.679.
  • Eighteen of the thirty correlations of CHI-B with non-corresponding HEXACO-60 domains were not significant.
  • Differences between CHI-B and BHI in the expected associations with authoritarianism and SDO were not significant.
  • Domain reliabilities ranged between 0.45 and 0.65 for CHI-R and between 0.22 and 0.62 for CHI-V.
  • The mean alpha of CHI-R was 0.56 versus 0.51 for CHI-B; chi-square(1)=0.41 and p=0.734.
  • The mean alpha of CHI-V was 0.50 versus 0.51 for CHI-B; chi-square(1)=0.01 and p=0.908.
  • CHI-V significantly reduced the alpha of Emotionality and Conscientiousness relative to CHI-B.
  • The mean convergent correlation was 0.66 for CHI-R and 0.63 for CHI-V, with no improvement over the 0.68 of CHI-B.
  • Mean absolute discriminant correlations were 0.10 for CHI-R and CHI-V and did not improve over the 0.12 of CHI-B.
  • Only 2 of 35 contrasts between CHI-R and BHI were significant after FDR.
  • Only 3 of 35 contrasts between CHI-V and BHI were significant and all three corresponded to Emotionality.
  • The alpha of Emotionality was 0.22 in CHI-V versus 0.50 in BHI; chi-square(1)=7.76 and p=0.037.
  • The convergence of Emotionality was 0.50 in CHI-V versus 0.75 in BHI; z=4.96 and p<0.001.
  • The Emotionality-authoritarianism correlation was -0.19 in CHI-V and 0.01 in BHI; z=2.81 and p=0.035.
  • A negative key error left Honesty-Humility without reverse-scored items in all three versions and another did the same for Agreeableness in CHI-V.
  • General instructions to optimize reliability or content did not consistently improve the measured properties.
  • The result refers to a questionnaire for people generated with the help of ChatGPT, not to the personality of ChatGPT.

Limitations

  • The study is declared exploratory.
  • No preregistration is reported.
  • No power calculation or prospective justification of sample size is provided.
  • Recruitment comes from the personal networks of undergraduate students.
  • The convenience sample does not necessarily represent the Dutch population.
  • Potential dependence among participants recruited by the same students is not modeled.
  • The gender distribution is unbalanced and limited to three aggregated categories.
  • Validation was conducted only in Dutch.
  • Linguistic equivalence with the original English items is not studied.
  • No formal back-translation procedure is described.
  • The exact date on which ChatGPT was used is not reported.
  • The label ChatGPT 4.0 does not identify a reproducible snapshot.
  • No system prompt, temperature, top-p, seed, or session configuration is published.
  • Only one final generation per version of the questionnaire is retained.
  • Variability across multiple generations of the same prompt is not quantified.
  • Other generative models are not compared.
  • The workflow is not repeated on different dates or accounts.
  • The authors had to correct two English wording problems.
  • The authors had to correct eight translation problems.
  • The need for corrections demonstrates material human contribution.
  • Two negative key errors survived final review.
  • Honesty-Humility was left without negative items in all three versions.
  • Agreeableness was left without negative items in CHI-V.
  • No blind independent review of the items is reported.
  • No formal expert panel for content validity is presented.
  • No content validity index or ratio is calculated.
  • No cognitive interviews with participants are conducted.
  • No qualitative pilot prior to final administration is reported.
  • No exploratory factor analysis is performed.
  • No confirmatory factor analysis is performed.
  • The six-factor structure is not tested.
  • Invariance across gender, age, or other groups is not evaluated.
  • Item response theory is not applied.
  • Test-retest reliability is not studied.
  • No omega or estimators other than alpha are reported.
  • Alpha may be low by design in broad domains measured with only four items.
  • Low reliabilities attenuate and make validity correlations more uncertain.
  • The alpha of authoritarianism is only 0.48.
  • Only authoritarianism and SDO are used as external criteria.
  • No behavioral criteria or observable outcomes are included.
  • All measures are self-reports completed in a single session.
  • Concurrent correlations do not prove predictive validity.
  • The operationalization of content validity depends mainly on correlations, not on expert coverage of the construct.
  • The instruction to optimize content does not specify a verifiable quantitative criterion.
  • The instruction to optimize reliability does not set a target alpha, such as 0.70.
  • No equivalence or non-inferiority design is used.
  • The absence of a significant difference does not demonstrate that CHI and BHI are equivalent.
  • The CHI-R and CHI-V samples are different from the CHI-B sample.
  • Randomization reduces confounding between groups, but no baseline balance table is published.
  • The sizes of the three groups are unequal.
  • Random assignment and its concealment are not detailed.
  • The distribution of the eleven exclusions across groups is not reported.
  • The treatment of missing data is not described beyond participants completing the measures.
  • Numerous tests are performed and FDR correction does not eliminate the risk of isolated findings.
  • Averaging the six domains may mask important deteriorations such as Emotionality in CHI-V.
  • Comparisons focus on significance and do not publish equivalence intervals.
  • Participant-level data are not published.
  • Data are only available upon request to the authors.
  • No analysis code is published.
  • No complete reproducible workflow from prompts to tables is provided.
  • ChatGPT's own statement about its training data is not verifiable.
  • Prior exposure of the model to HEXACO, BHI, IPIP, or other item banks cannot be ruled out.
  • The fact that the final items do not literally overlap with the questionnaires used does not rule out influence of training material.
  • The official supplement contains prompts and items, but the publisher prevented freezing it outside the page during this audit.
  • The review does not attribute exact prompt formulations that are not in the main body.
  • Time, cost, or savings relative to human development are not quantified.
  • The process is not compared with human teams under equivalent resources.
  • There is no independent replication of the questionnaire.
  • CHI is not validated in a second sample.
  • Stability under subsequent ChatGPT updates is not examined.
  • Results from a 2024 product cannot be automatically generalized to current models.
  • The study does not separate which part of the result comes from the model and which from human corrections.
  • The open license of the article does not imply that the underlying data are open.

What the study does not establish

  • It does not demonstrate that ChatGPT has personality or HEXACO traits.
  • It does not measure ChatGPT's responses to a personality questionnaire.
  • It does not demonstrate that ChatGPT can create a valid scale without human supervision.
  • It does not demonstrate psychometric equivalence between CHI and BHI.
  • It does not demonstrate superiority of CHI over a human scale.
  • It does not demonstrate that requesting higher reliability actually increases reliability.
  • It does not demonstrate that requesting higher content validity actually increases that validity.
  • It does not validate the factor structure of any CHI version.
  • It does not demonstrate invariance across demographic groups or languages.
  • It does not establish temporal reliability of CHI scores.
  • It does not demonstrate predictive validity with respect to external behaviors.
  • It does not demonstrate that the results replicate with another sample or generative model.
  • It does not rule out that ChatGPT knew personality materials during training.
  • It does not test that the workflow reduces time or cost relative to expert development.
  • It does not allow exact reproduction of the generation without snapshot, configuration, and frozen supplement.

Traceability

Scope: Full text

Version: Journal of Personality Assessment 107(4):419–425; published online 27 Dec 2024; publisher version; CC BY 4.0

Consulted source: https://scholarlypublications.universiteitleiden.nl/access/item%3A4250852/view

Review: Codex full-text, visual, bilingual-fidelity, psychometric-validity, statistical-interpretation, item-generation, source-transparency and claim-boundary audit, 2026-07-15

Approval: Codex fidelity pass, 2026-07-15

English translation: approved, 2026-07-18

Models evaluated

  • ChatGPT 4.0 through the public ChatGPT product; exact model snapshot, date and generation configuration not reported

Instruments and metrics

  • Dutch HEXACO-60: 60 items, ten per domain, five-point response scale
  • Dutch Brief HEXACO Inventory (BHI): 24 human-written items, four per domain
  • ChatGPT HEXACO Inventory baseline (CHI-B): 24 generated items, four per domain
  • Reliability-enhanced ChatGPT HEXACO Inventory (CHI-R): 24 rewritten items
  • Content-validity-enhanced ChatGPT HEXACO Inventory (CHI-V): 24 rewritten items
  • Child-rearing values authoritarianism scale: four forced choices
  • Social Dominance Orientation SDO7 short scale: eight items on seven points
  • Cronbach alpha reliability comparisons using the Diedenhofen and Musch procedure
  • Convergent and discriminant correlations with HEXACO-60
  • Criterion-related correlations with authoritarianism and SDO
  • Benjamini-Hochberg False Discovery Rate correction
  • HEXACO-60 noncompliant-response and average response-speed exclusions

Data used

  • Final human sample of 682 participants
  • CHI-B randomized subgroup, n=228
  • CHI-R randomized subgroup, n=242
  • CHI-V randomized subgroup, n=212
  • Published aggregate results in Tables 1–2
  • Supplemental Tables S1–S5 referenced by the article but not frozen during this audit
  • Underlying participant data available from the authors only on request

Evidence and location

  • Bibliographic identity, dates, license, and pagination: Journal of Personality Assessment 107(4):419-425; DOI 10.1080/00223891.2024.2444454; online 27 Dec 2024; CC BY 4.0
  • Full editorial PDF inspected: .cache/editorial-sources/article-080/source.pdf; Leiden University repository; sha256 f128f9133fc9160aa73791c631c2283527a5a3b1ae7423e8b88b56f257b8c1c8
  • Complete official abstract: Publisher article, abstract, journal p. 419
  • Sample, exclusions, and demographics: Methods, Sample, journal p. 420
  • Procedure and assignment to CHI versions: Methods, Procedure, journal p. 420
  • Generation, rules, corrections, and translation of CHI: Methods, CHI versions, journal pp. 420-421
  • Negative key errors: Footnote 2, journal p. 419
  • Instruments and reference reliabilities: Methods, Materials, journal pp. 420-421
  • Comparison of CHI-B versus BHI: Results and Table 1, journal pp. 421-422
  • Comparison of the three CHI versions: Table 2 and Comparing the CHI versions, journal pp. 422-423
  • Comparisons of CHI-R and CHI-V versus BHI: Results, journal p. 423
  • Interpretation and recognized limits: Discussion, journal pp. 423-424
  • Data not open: Data availability statement, journal p. 424: Data is available on request
  • Official supplement identified but not frozen: Taylor & Francis Supplemental tab: ChatGPT HEXACO supplemental files R2.docx, 88.7 KB; download returned HTTP 403 during audit on 15 Jul 2026
  • Comprehensive reading and visual inspection: All 9 repository and publisher PDF pages rendered and inspected, including Tables 1-2, footnotes, Discussion and references; checked 15 Jul 2026