Applying Psychometrics to Large Language Model Simulated Populations: Recreating the HEXACO Personality Inventory Experiment with Generative Agents

Society, culture, and collective behavior2025arXivApproved editorial review

Authors: Sarah Mercer, Daniel P. Martin, Phil Swatton

Keywords: Computation and Language, Machine Learning, Generative Agents, Personality Inventory, Psychometrics, HEXACO

Source: Open primary source (opens in a new tab)

3
Authors
7
Findings
9
Limitations
9
Evidence

Editorial summary

English

The paper asks whether a population of biography-conditioned generative agents can produce a lexical personality structure comparable to HEXACO. GPT-4 creates 310 biographies and each agent rates 1,710 adjectives on a nine-point scale; responses are ipsatized and analyzed with principal components and promax rotation. The PopCensus population matches the 2021 occupational distribution of England and Wales, but it does not represent that population's age, gender, ethnicity, education, or geography. The scree plot suggests five factors. The five-factor solution explains 19.54% of variance and the six-factor solution 19.71%, whereas the authors select ten factors, explaining 22.51%, because that solution has the highest mean Cronbach's alpha without reversing items whose loadings have opposite signs. Auditing the open code and data reproduces the reported alphas but shows that the implementation mixes item polarities: orienting each factor's 30 selected items by loading sign changes mean alpha from 0.219 to 0.973 for five factors, from 0.287 to 0.972 for six, and from 0.728 to 0.959 for ten. The published negative alphas are therefore largely coding artifacts, and the criterion used to prefer ten factors is not robust; the corrected high alphas do not themselves establish validity because item selection and evaluation use the same sample. Exact lexical overlap with HEXACO is very small, although a FastText comparison finds greater semantic association. A second test administers the HEXACO-PI-R-100 to the same biographies: mean absolute correlations with the lexical factors are 0.765 for GPT-4, 0.751 for Claude 3.7 Sonnet, 0.712 for Phi-4, and 0.217 for Llama 3.2. This indicates that larger models reuse signals from a synthetic persona consistently, but it does not establish equivalence with human traits because both measurements derive from the same biographies and learned stereotypes. A second professional population fails to recover Honesty-Humility, showing sensitivity to persona wording and composition. The paper appropriately interprets its factors as linguistic associations rather than internal personality. Releasing responses, processed data, and notebooks is a major strength, although the persona- and response-generation pipeline, exact dependency versions, and precise GPT-4 snapshot are absent.

Español

El trabajo pregunta si una población de agentes generativos condicionados por biografías puede producir una estructura léxica de personalidad comparable con HEXACO. GPT-4 crea 310 biografías y hace que cada agente valore 1.710 adjetivos en una escala de nueve puntos; las respuestas se ipsatizan y se someten a análisis de componentes principales con rotación promax. La población PopCensus reproduce la distribución de ocupaciones de Inglaterra y Gales de 2021, pero no representa su distribución de edad, género, etnia, educación o geografía. El scree plot sugiere cinco factores. Esa solución explica el 19,54 % de la varianza y la de seis factores el 19,71 %, mientras que los autores eligen diez factores, con un 22,51 %, porque obtiene el mayor alfa de Cronbach medio sin invertir los ítems cuyas cargas tienen signo opuesto. La auditoría del código y los datos abiertos reproduce los alfas publicados, pero muestra que esa implementación mezcla polaridades: al orientar los 30 ítems de cada factor según el signo de su carga, el alfa medio pasa de 0,219 a 0,973 en cinco factores, de 0,287 a 0,972 en seis y de 0,728 a 0,959 en diez. Por tanto, los alfas negativos publicados son en gran medida artefactos de codificación y el criterio usado para preferir diez factores no es robusto; los alfas corregidos tampoco prueban validez, porque los ítems se seleccionan y evalúan en la misma muestra. Los solapamientos léxicos exactos con HEXACO son muy pequeños, aunque una comparación semántica con FastText encuentra asociaciones mayores. Una segunda prueba hace responder el HEXACO-PI-R-100 a las mismas biografías: las correlaciones absolutas medias con los factores léxicos son 0,765 para GPT-4, 0,751 para Claude 3.7 Sonnet, 0,712 para Phi-4 y 0,217 para Llama 3.2. Esto indica que los modelos grandes reutilizan de manera consistente las señales de una persona sintética, pero no demuestra equivalencia con rasgos humanos, pues las dos mediciones parten de las mismas biografías y de estereotipos aprendidos. Una segunda población profesional no recupera Honesty-Humility, lo que evidencia sensibilidad a la redacción y composición de las personas. El artículo es prudente al interpretar los factores como asociaciones lingüísticas, no como personalidad interna. La publicación de respuestas, datos procesados y notebooks es una fortaleza importante, aunque faltan el pipeline que generó personas y respuestas, versiones exactas de dependencias y la instantánea precisa de GPT-4.

Research question

Can a synthetic population of GPT-4 agents conditioned by biographies reproduce a lexical personality structure comparable with HEXACO, maintain internal consistency, and converge with responses to a HEXACO inventory across different models?

Method

Computational psychometric study in three blocks. First, GPT-4 generates PopCensus and each of its 310 agents rates 1,710 adjectives with explanation; after parsing, ipsatization, and removal of items with no variance, PCA and promax rotation are applied, solutions of five to twelve factors are inspected, and alpha, lexical overlap, and FastText similarity are calculated. Second, GPT-4, Claude 3.7 Sonnet, Phi-4 14B, and Llama 3.2 3B respond to HEXACO-PI-R-100 for the same persons and their domains are correlated with the lexical factor scores. Third, the procedure is replicated with PopProfessional to study sensitivity to another composition of persons. The editorial review reproduced the results from the repository data and notebooks and recalculated reliability by orienting the items by the sign of their loading.

Sample: PopCensus contains 310 biographies, 272 unique occupations, and ages from 18 to 60 years; only the occupational distribution aligns with the census of England and Wales. The text reports 200 male agents and 102 female agents, so eight remain unexplained in that count. Each agent produces 1,710 ratings. The PopProfessional replication contains 313 persons and 83 unique occupations, despite parts of the article describing it as a population of 310.

Findings

  • The scree plot favors five factors; the five-, six-, and ten-factor solutions explain respectively 19.54%, 19.71%, and 22.51% of the variance. The ten factors receive post hoc labels such as Dishonesty, Disagreeableness, Introversion, and Unconscientiousness.
  • The published alphas are obtained by selecting the 30 items with the highest absolute loading and calculating alpha on their values without orienting the signs. This reproduces mean alphas of 0.219, 0.287, and 0.728 for five, six, and ten factors; after reversing the polarity indicated by the loadings, they rise to 0.973, 0.972, and 0.959.
  • The correction demonstrates that the negative alphas do not evidence internally incoherent factors, but it also removes the basis for choosing ten factors for their higher raw alpha. Item selection and reliability estimation on the same sample make the corrected alphas optimistic.
  • The exact overlap of each synthetic factor with a HEXACO factor only ranges between 0.024 and 0.166. The metric called weighted Jaccard uses membership and sign, not the magnitude of the loadings; the FastText comparison ignores sign and may treat antonyms as semantically close.
  • The mean absolute correlations between GPT-4 lexical factors and HEXACO-PI-R-100 domains are 0.765 with GPT-4, 0.751 with Claude 3.7 Sonnet, 0.712 with Phi-4, and 0.217 with Llama 3.2; the three larger models reuse biographical signals better than the 3B model.
  • PopProfessional obtains eight factors and a mean raw alpha of 0.660, but does not recover Honesty-Humility. Its occupational composition and the use of less intense negative facts change the structure, showing dependence on the design of persons and prompts.
  • The consistency based on 342 pairs derived from prefixes reaches a per-agent mean of 0.812 and 83.04% of the pairs exceed 0.75. The partial length of the biography correlates r = 0.395 with that consistency, a moderate association that does not identify a causal effect.

Limitations

  • The population is synthetic and only calibrated by occupation; its age does not align with the census and ethnicity, education, geography, or a complete gender distribution are not controlled. Calling it representative without that nuance would exaggerate the design.
  • GPT-4 generates the biographies and also the lexical responses, so the structure may reflect dependencies of the same generator, narrative conventions, stereotypes, and safety restrictions.
  • There are about 310 observations versus around 1,700 variables, with no parallel analysis, bootstrap, holdout, or cross-validation; inspecting from five to twelve factors and choosing by alpha on the same sample favors overfitting.
  • The comparison of variance between ten synthetic factors and six human factors is not equivalent, because adding components necessarily increases the explained variance.
  • The convergent correlations reuse the same biographies in both instruments. They may measure consistency by extracting facts or stereotypes about the person, not an independent latent construct.
  • The parser substitutes 5 for uninterpretable responses. In PopCensus there are 291 errors, 225 associated with niggardly and 66 distributed among other adjectives; this neutral imputation is not clearly described in the article.
  • The semantic similarity ignores polarity, the random baseline uses only ten samples without a seed, and its published mean, 0.363, does not match the current repository output, 0.378.
  • The repository provides data and analysis, but not the code that generated biographies or queried the models, nor a lockfile, exact dependency versions, complete seeds, or a GPT-4 snapshot identifier; reproduction of the collection is incomplete.
  • The factor names are post hoc human interpretations without independent evaluators, and there is no external behavioral or longitudinal test that validates those meanings.

What the study does not establish

  • It does not demonstrate that GPT-4, Claude, Phi, or Llama possess personality, internal traits, identity, emotions, or human motivations.
  • It does not demonstrate that PopCensus represents the population of England and Wales beyond its approximate occupational distribution.
  • It does not unambiguously recover the HEXACO structure: the number of factors, the labels, the lexical overlap, and the disappearance of Honesty-Humility in PopProfessional depend on the analytical design and on the persons.
  • It does not validate the direct use of human psychometric instruments with LLMs nor establish measurement invariance or construct equivalence between agents and persons.
  • It does not prove that the correlations between the lexical inventory and HEXACO are independent of the shared biography, the generating model, or learned stereotypes.
  • It does not establish that ten factors are the correct or most reliable solution, due to the item orientation error and the absence of out-of-sample validation.
  • It does not evaluate safety, manipulation, user well-being, discrimination, or agent behavior in real applications.

Traceability

Scope: Full text

Version: arXiv:2508.00742v2 (22 Sep 2025); linked repository commit e6a25d2b720f87825bfa57cdad425e12d288909a also audited

Consulted source: https://arxiv.org/pdf/2508.00742

Review: Codex editorial review and reproducibility audit, 2026-07-14

Approval: Codex fidelity pass, 2026-07-14

English translation: approved, 2026-07-18

Models evaluated

  • GPT-4 (deployment or snapshot not specified)
  • Claude 3.7 Sonnet
  • Phi-4 14B
  • Llama 3.2 3B

Instruments and metrics

  • 1,710-adjective lexical personality inventory
  • HEXACO-PI-R-100
  • Principal component analysis with promax rotation
  • Cronbach's alpha
  • Exact and sign-sensitive weighted Jaccard comparisons
  • FastText semantic similarity
  • Prefix-derived antonym-pair consistency measure

Data used

  • PopCensus synthetic biographies and adjective ratings
  • PopProfessional synthetic biographies and adjective ratings
  • 2021 England and Wales census occupational distribution
  • HEXACO lexical adjective data reconstructed from the available Harvard Dataverse substitute
  • Released HEXACO questionnaire responses from four LLMs

Evidence and location

  • Design, construction of PopCensus, prompts, sample, and inventory of 1,710 adjectives: Paper, pp. 3–6, sections 2.1–2.4 and Appendix A
  • Scree plot, factor solutions, variance, alphas, and selection of ten factors: Paper, pp. 6–9, section 3.1, Figures 2–3 and Tables 2–4
  • Overlap with HEXACO, FastText similarity, and random baseline: Paper, pp. 9–11, sections 3.2–3.3 and Figures 4–5
  • Convergence between models using HEXACO-PI-R-100: Paper, pp. 11–13, section 3.4 and Table 1; repository notebooks and released response data at commit e6a25d2b720f87825bfa57cdad425e12d288909a
  • Consistency, prefix-derived pairs, and association with biographical length: Paper, pp. 13–14, section 3.5 and Figure 6; repository analysis helpers at audited commit
  • PopProfessional replication and sensitivity of Honesty-Humility: Paper, pp. 14–16, section 3.6 and Figure 7; released PopProfessional data at audited commit
  • Interpretation, limitations, and caution against attributing human personality: Paper, pp. 16–19, sections 4–6
  • Alpha orientation error and recalculated results: Linked repository commit e6a25d2b720f87825bfa57cdad425e12d288909a, results_template.py and support/solution_support.py; independent reproduction from released PopCensus and PopProfessional matrices
  • Parsing errors, neutral imputation, and reproducible pipeline shortcomings: Linked repository commit e6a25d2b720f87825bfa57cdad425e12d288909a, raw response files, parser defaults, README and full git history