Can LLMs Assess Personality? Validating Conversational AI for Trait Profiling

Evaluation and psychometric validity2026arXivApproved editorial review

Authors: Andrius Matšenas, Anet Lello, Tõnis Lees, Hans Peep, Kim Lilii Tamm

Keywords: Personality, Persona conditioning, Psychometrics

Source: Open primary source (opens in a new tab)

5
Authors
8
Findings
25
Limitations
4
Evidence

Editorial summary

English

The preprint asks whether a guided conversation can produce Big Five profiles comparable to IPIP-50. Thirty-three convenience participants first complete twenty brief conversations with Gemini 2.5 Flash, four questions per trait and usually zero or one follow-up, and then answer IPIP-50. A separate LLM component receives all twenty dialogues, assigns 0-120 trait scores and confidence, and the system normalizes both methods to 0-100. Participants then rate the perceived accuracy of each result and choose their preferred method. Method order, trait blocks, and questions are fixed; there is no counterbalancing, blinding, false profile, human interviewer, or nonadaptive conversational condition.

The paper reports conversation-IPIP correlations of .38 for Agreeableness, .42 for Conscientiousness, .58 for Extraversion, .44 for Neuroticism, and .46 for Openness. At N=33, recalculated Fisher intervals are wide, approximately [.042,.640], [.090,.667], [.296,.770], [.114,.681], and [.139,.694], and observed shared variance is only 14%-34%. Paired score comparisons give p=.002, .036, .084, .485, and .441 for Agreeableness, Extraversion, Conscientiousness, Openness, and Neuroticism; perceived-accuracy comparisons give p=.745, .144, .297, .586, and .686. Participants rate both approaches around 4/5 and most prefer conversation, but exact means, standard deviations, differences, intervals, and paired observations are not published.

The abstract's equivalence claim is unsupported: no TOST or other equivalence procedure is run, there is no margin, and a nonsignificant result does not demonstrate equality. After Bonferroni across the five score tests, only the Agreeableness difference remains significant at alpha=.01; Extraversion does not. The table provides p-values alone, without t statistics, degrees of freedom, effects, intervals, or diagnostics. The study also reports no IPIP reliability in this sample, test-retest assessment, repeated scorer runs, factor or discriminant validity, calibration, or external criterion. Accuracy ratings are unblinded impressions vulnerable to demand and Barnum effects. PCA appears to use roughly 21 variables for 33 people, and clustering omits scaling, distance, linkage, and stability, so its profiles and age patterns remain exploratory.

No data, transcripts, code, outputs, analysis scripts, call logs, preregistration, or public artifact is available; the model snapshot, parameters, normalization formula, reverse scoring, and missingness handling are also unspecified. The ethics appendix describes consent, restricted access, and identifier deletion, but does not name an ethics committee or approval and does not explain how potentially sensitive narratives were processed through Gemini or how its no-third-party-PII statement applies. The defensible contribution is a small pilot showing moderate association and favorable acceptance of a conversational interface for eliciting self-description. It does not validate replacement of IPIP-50, measurement of observed behavior, equivalence between methods, or deployment in hiring, therapy, education, dating, or other high-impact decisions.

Español

El preprint estudia si una conversación guiada puede producir perfiles Big Five comparables con el IPIP-50. Treinta y tres participantes de conveniencia completan primero veinte conversaciones breves con Gemini 2.5 Flash, cuatro preguntas por rasgo y normalmente cero o un follow-up, y después el cuestionario IPIP-50. Otro componente LLM recibe los veinte diálogos, asigna scores 0-120 y confianza a los cinco rasgos y el sistema normaliza ambos métodos a 0-100. Al final, cada participante valora la precisión percibida de cada resultado y elige su método preferido. El orden de métodos, bloques de rasgo y preguntas es fijo; no hay contrabalanceo, cegamiento, perfil falso, entrevistador humano ni condición conversacional no adaptativa.

El paper reporta correlaciones entre los scores conversacionales e IPIP de 0,38 en amabilidad, 0,42 en responsabilidad, 0,58 en extraversión, 0,44 en neuroticismo y 0,46 en apertura. Con N=33, sus intervalos de Fisher recalculados son amplios, aproximadamente [.042,.640], [.090,.667], [.296,.770], [.114,.681] y [.139,.694], y la varianza compartida observada queda entre 14% y 34%. Las comparaciones pareadas de score dan p=.002, .036, .084, .485 y .441 para amabilidad, extraversión, responsabilidad, apertura y neuroticismo; las de precisión percibida dan p=.745, .144, .297, .586 y .686. Los participantes puntúan ambos métodos alrededor de 4/5 y la mayoría prefiere conversar, pero no se publican medias exactas, desviaciones, diferencias, intervalos o datos emparejados.

La afirmación de equivalencia del abstract no está sustentada: no se ejecuta TOST ni otra prueba de equivalencia, no existe margen y un resultado no significativo no demuestra igualdad. Tras Bonferroni para los cinco tests de score, solo la diferencia de amabilidad conserva significación a alfa=.01; extraversión no. La tabla solo muestra p-values, sin t, grados de libertad, efectos, intervalos ni diagnósticos. Tampoco se reportan fiabilidad del IPIP en esta muestra, test-retest, repetición del scorer, validez factorial o discriminante, calibración o criterio externo. Las ratings de precisión son impresiones no cegadas, expuestas a demanda y efecto Barnum. El análisis PCA usa aproximadamente 21 variables con 33 personas y el clustering omite escalado, distancia, linkage y estabilidad, por lo que sus perfiles y asociaciones con edad son exploratorios.

No hay datos, transcripciones, código, salidas, scripts, logs de llamadas, preregistro ni artefacto público; tampoco se especifican snapshot del modelo, parámetros, fórmula de normalización, reverse scoring o tratamiento de missingness. El anexo ético declara consentimiento, acceso restringido y eliminación de identificadores, pero no identifica comité o aprobación ni explica el procesamiento de narrativas potencialmente sensibles por Gemini o cómo se aplica la afirmación de no compartir PII con terceros. La contribución defendible es un piloto pequeño que muestra asociación moderada y buena aceptación de una interfaz conversacional para recabar autodescripciones. No valida sustitución del IPIP-50, medición de conducta observada, equivalencia entre métodos ni uso en contratación, terapia, educación, citas u otras decisiones de alto impacto.

Research question

To what extent do Big Five scores inferred by an LLM from twenty brief conversations converge with the IPIP-50 and are perceived by the same participants as equally accurate or preferable?

Method

Within-subject design with fixed order in 33 convenience participants: twenty brief conversations, four for each Big Five, with Gemini 2.5 Flash; joint LLM scoring of the transcripts on a 0-120 scale; subsequent IPIP-50; normalization to 0-100; 1-5 ratings of perceived accuracy and global preference. Analysis via ten paired t-tests, five homologous correlations, PCA and hierarchical clustering.

Sample: Thirty-three complete participations recruited through social networks and university channels; initial target 100 and absolute minimum 30. The text describes a predominantly young, educated and technology-familiar sample, but does not publish a demographic table, number of starts, dropouts, exclusions, compensation or participant flow.

Findings

  • The homologous conversation-IPIP correlations range from r=.38 to r=.58 and share approximately 14%-34% of observed variance.
  • The confidence intervals recalculated for N=33 are wide and compatible with associations considerably lower than the point estimates.
  • The p-values for score difference are .002, .036, .084, .485 and .441 for agreeableness, extraversion, conscientiousness, openness and neuroticism.
  • With Bonferroni for five score tests, only agreeableness remains significant at alpha=.01.
  • The five perceived accuracy tests are non-significant, but this does not demonstrate equivalence of perception.
  • Both methods receive ratings close to 4/5 and the majority declare preferring the conversation.
  • The study reports PCA with 32.45%, 17.38% and 12.88% in the first three components, 62.70% combined.
  • The results support preliminary convergence and interface acceptance, not psychometric interchangeability.

Limitations

  • Convenience sample of N=33, far below the target of 100 and without a priori power based on an effect.
  • No participant flow, attrition, exclusions or quantified demographics are published.
  • Conversation always before the IPIP and trait blocks always in the same order.
  • There is no randomization, counterbalancing, blinding or controls for false or generic profiles.
  • There is no baseline of fixed open-ended questions that would allow isolating the value of adaptive follow-up.
  • Appendix A limits the exchange to one or two participant messages, while Methods allows up to ten.
  • The general purpose of AI evaluation is revealed before responding and may induce demand.
  • The scorer observes self-descriptions, not external and independent behavior.
  • The exact identity of the scorer is unknown and a snapshot, date, parameters, seeds, retries and safety controls are missing.
  • The 0-120 scale and its thresholds are defined by the study and are not psychometrically justified.
  • The normalization formulas, reverse scoring or treatment of missing data are not published.
  • No alpha, omega, factor structure or measurement error of the IPIP-50 in the sample is reported.
  • There is no test-retest, scoring repetition, inter-model reliability or comparison with a human evaluator.
  • The homologous correlations are not accompanied by a cross-trait matrix or discriminant validity.
  • There is no external behavioral criterion, calibration or predictive validity.
  • The ten t-tests have no multiplicity control and Table 1 publishes only p-values.
  • Absence of significant difference is incorrectly interpreted as equivalence.
  • The 1-5 ratings are analyzed with a t-test without distribution, effects or intervals and measure perception, not objective accuracy.
  • There is no control for Barnum/Forer effects, social desirability or preference for a more attractive interface.
  • PCA with about 21 variables and 33 people lacks ratio and sufficient reporting for a stable structure.
  • Clustering omits distance metric, linkage, scaling, stability and external validation.
  • Comparisons with previous work mix instruments, samples, texts and distinct models without formal testing.
  • Fairness, subgroup bias, adverse impact or safety of high-risk uses are not evaluated.
  • No identifiable ethical approval or exemption or detail of narrative processing by the provider is published.
  • There are no public data, transcripts, code, analyses, outputs, logs, preregistration or artifacts.

What the study does not establish

  • It does not establish statistical equivalence or interchangeability between conversation and IPIP-50.
  • It does not validate the system as a substitute for professional psychometric evaluation.
  • It does not establish that the LLM measures observed behavior; it interprets conversational self-descriptions.
  • It does not demonstrate that dynamic follow-up improves the profile over non-adaptive open-ended questions.
  • It does not demonstrate reliability, factorial, discriminant, predictive or incremental validity.
  • It does not demonstrate clinical, occupational, educational or relational accuracy or safety in high-impact decisions.
  • It does not generalize to representative populations, languages, cultures, models or different configurations.
  • It does not allow reproducing the quantitative results with public materials.
  • It does not resolve privacy, governance or fairness of automated analysis of personal narratives.

Traceability

Scope: Full text

Version: arXiv:2602.15848v1, submitted and last updated 2026-01-23; CC BY 4.0

Consulted source: https://arxiv.org/abs/2602.15848

Review: Codex 13-page visual full-text, complete arXiv source, conversational design, IPIP comparison, equivalence, multiplicity, correlation uncertainty, scale, reliability, PCA, clustering, privacy, artifact and claim-boundary audit, 2026-07-18

Approval: Codex fidelity pass, 2026-07-18

English translation: approved, 2026-07-18

Models evaluated

  • Gemini 2.5 Flash como componente conversacional, sin identificador inmutable, fecha de servicio ni configuración completa
  • LLM de scoring no identificado de forma inequívoca, sin snapshot, parámetros ni evaluación de repetibilidad

Instruments and metrics

  • Veinte preguntas conversacionales fijas, cuatro por cada rasgo Big Five
  • Follow-up dinámico, descrito como máximo uno en Appendix A pero hasta diez mensajes de usuario en Methods
  • Prompt de scoring Big Five con facetas, evidencia textual, confianza y escala 0-120
  • International Personality Item Pool IPIP-50, diez ítems de cinco puntos por rasgo
  • Ratings 1-5 de precisión percibida por rasgo y método
  • Preferencia global entre conversación e IPIP-50
  • t-tests pareados sin corrección por multiplicidad
  • Correlaciones Pearson homólogas sin matriz discriminante
  • PCA y clustering jerárquico exploratorios

Data used

  • Datos de 33 participantes descritos pero no publicados
  • Veinte transcripciones conversacionales por participante no publicadas
  • Scores IPIP, scores LLM, ratings y preferencia individuales no publicados
  • No se localizó repositorio, dataset, preregistro o suplemento público

Evidence and location

  • Design, procedure, questions, scoring, results, figures, tables, discussion and appendices: arXiv:2602.15848v1, all 13/13 PDF pages rendered and individually inspected
  • Version, date, authorship, categories and license: Official arXiv abstract and Atom metadata inspected 2026-07-18
  • Complete source, printed prompts and absence of hidden artifacts: Complete official arXiv v1 source archive sha256 d64000b866ca3556f66021c63f83fe8126248e43071e5ad6e4a0fad7d3c33d59; main.tex sha256 ca6c420e0ee8df7afd1190d87e8ed133a32290641ce85a40079334ece4aec199
  • Equivalence, multiplicity, correlations, scales, reliability, order, ethics, privacy, artifacts and limits audit: reports/verification/article-402-conversational-selfreport-equivalence-multiplicity-scale-reliability-privacy-artifact-and-claim-audit.json