Synthetic Personality Review

Personality-aware Student Simulation for Conversational Intelligent Tutoring Systems

Applications, bias, and safety2024ACL AnthologyApproved editorial review

Authors: Zhengyuan Liu, Stella Xin Yin, Geyu Lin, Nancy F. Chen

Keywords: intelligent tutoring systems, student simulation, personality traits, educational technology, language learning, conversational AI, personalized learning

Source: Open primary source (opens in a new tab)

4
Authors
19
Findings
38
Limitations
18
Evidence

Editorial summary

English

The paper proposes a student simulator for language-learning conversations that combines two language-ability levels with a contextual Big Five adaptation called BF-TC. High versus low ability is defined through five Narrative Assessment Protocol components: phrases, sentence structure, modifiers, nouns, and verbs. BF-TC turns each trait into high/low tutoring behaviors: for example, openness includes creativity and interest in learning; conscientiousness includes organization, attitude, and strategies; extraversion includes participation and hesitation; agreeableness includes interest, empathy, and politeness; and neuroticism includes anxiety, mood shifts, and confidence. Zephyr-7B-beta, Vicuna-13B-v1.5, GPT-3.5-1106, and GPT-4-1106 play both teacher and student in image-description dialogues, with the same model filling both roles. The study reports 100 open-source cartoon images, 500 synthetic dialogues, and 10,000 utterances. GPT-4 is the common judge for BF-TC classification, language-ability labeling, a generated 44-item BFI, and seven teacher-scaffolding labels. On 50 dialogues, two experts supply labels for a judge check: reported GPT-4 accuracies range from 0.78 for openness to 0.92 for extraversion and neuroticism, but the paper gives no inter-expert agreement, intervals, or adjudication protocol. For BF-TC classification, GPT-4 reaches 0.727 average F1; other generators range from 0.515 to 0.562, while three-shot prompting for Zephyr helps some traits and harms others. GPT-4 reaches 0.741 F1 for language ability. Generated BFI responses yield Cronbach’s alpha values from 0.906 to 0.936, and agreement between BF-TC and BFI labels averages 0.725 to 0.802 F1 across generators. Correlations also appear between imposed student profiles and model-labeled teacher scaffolding: in this synthetic sample, the low-ability condition receives more hints, explanations, and modeling, whereas the high-ability condition receives more feedback, instructions, and questions. These results show that some LLMs, especially GPT-4-1106, can follow prompts that create distinguishable student styles and different teacher responses inside a model-generated, model-judged loop. They do not validate real students’ personalities, learning, pedagogical effectiveness, or an independent psychological measure. BF-TC directly embeds the cues later used for classification and blends traits with competence, motivation, compliance, and social desirability, so much of the agreement is expected by construction. There are no child participants, real teachers, pre/post learning outcomes, behavioral comparison with humans, or official releases of the code, images, dialogues, and annotations needed for reproduction.

Español

El artículo propone un simulador de alumnado para conversaciones de aprendizaje de idiomas que combina dos niveles de capacidad lingüística con una adaptación contextual del Big Five llamada BF-TC. La capacidad alta o baja se define a partir de cinco componentes del Narrative Assessment Protocol, frases, estructura oracional, modificadores, sustantivos y verbos. Para BF-TC, cada rasgo se convierte en descripciones conductuales altas o bajas dentro de una tutoría: por ejemplo, apertura incluye creatividad e interés por aprender; responsabilidad incluye organización, actitud y estrategias; extraversión incluye participación y vacilaciones; amabilidad incluye interés, empatía y cortesía; y neuroticismo incluye ansiedad, cambios de humor y confianza. Zephyr-7B-beta, Vicuna-13B-v1.5, GPT-3.5-1106 y GPT-4-1106 interpretan tanto al docente como al estudiante en diálogos de descripción de imágenes; el mismo modelo ocupa ambos papeles. El estudio informa de 100 imágenes de dibujos animados de fuente abierta, 500 diálogos sintéticos y 10.000 turnos. GPT-4 se usa como juez común para clasificar BF-TC, capacidad lingüística, respuestas al BFI de 44 ítems y siete estrategias de andamiaje. En 50 diálogos, dos expertos proporcionan etiquetas para contrastar al juez: las exactitudes comunicadas para GPT-4 van de 0,78 en apertura a 0,92 en extraversión y neuroticismo, sin acuerdo entre expertos, intervalos ni protocolo de resolución. En la clasificación BF-TC, GPT-4 obtiene F1 medio 0,727; los otros modelos quedan entre 0,515 y 0,562, y el few-shot de tres ejemplos aplicado a Zephyr mejora unos rasgos pero empeora otros. Para capacidad lingüística, GPT-4 alcanza F1 0,741. El BFI generado muestra alfas de Cronbach entre 0,906 y 0,936, y la concordancia entre etiquetas BF-TC y BFI alcanza F1 medio entre 0,725 y 0,802 según el generador. También aparecen correlaciones entre perfiles impuestos y etiquetas de andamiaje del docente: en la muestra sintética, la condición de menor capacidad recibe más pistas, explicaciones y modelado, mientras la de mayor capacidad recibe más feedback, instrucciones y preguntas. Estos resultados evidencian que ciertos LLM, sobre todo GPT-4-1106, pueden seguir prompts que producen estilos de estudiante distinguibles y respuestas docentes diferentes dentro de este circuito generado y juzgado por modelos. No validan personalidad de estudiantes reales, aprendizaje, eficacia pedagógica ni una medición psicológica independiente. La definición BF-TC incorpora directamente señales que luego se clasifican y mezcla rasgos con competencia, motivación, obediencia y deseabilidad social; por ello gran parte de la concordancia es esperable por construcción. No hay participantes infantiles, docentes reales, pretest/postest, comparación con comportamiento humano ni artefactos oficiales de código, imágenes, diálogos o anotaciones para reproducir el trabajo.

Research question

Can an LLM simulate in tutoring dialogues student profiles defined by two levels of linguistic ability and by high or low versions of the five BF-TC traits; are those profiles recoverable through automatic classification and BFI; and does the scaffolding of a teacher, also simulated, change according to the imposed profile?

Method

The authors define high and low linguistic ability through five components inspired by NAP and reformulate the Big Five as BF-TC, a list of observable behaviors in educational conversations. Separate prompts specify the roles of teacher and student, the task of describing an image, the teacher's knowledge-building approach, the ability level, and the BF-TC descriptions. A multi-agent environment alternates both roles using the same LLM. Four models generate dialogues from image descriptions; GPT-4 labels each conversation by BF-TC traits and ability, has the simulated student answer the 44-item BFI, and classifies each teacher turn into seven scaffolding types. Two experts label 50 dialogues for a partial check of the judge. The analysis reports precision, recall, and F1, statistics and Cronbach's alpha of the BFI, Pearson correlations among its scales, an embedding visualization, and correlations between binary profile variables and scaffolding labels.

Sample: The empirical sample is entirely synthetic: four families/versions of LLM represent teacher and student from 100 image descriptions and produce 500 dialogues with 10,000 turns. The article does not break down with sufficient precision how many dialogues correspond to each model, trait, pole, ability, or image, nor how the five traits were combined simultaneously. Two experts label 50 conversations for a limited check of the automatic judge. No children, students, teachers, psychologists, schools, or end users participate.

Findings

  • BF-TC transforms the five traits into explicit signals of educational conversation, separated into high and low poles.
  • Linguistic ability is reduced to two conditions that prescribe complete and correct responses versus words, incomplete phrases, and grammatical errors.
  • In each dialogue, teacher and student are instances of the same generative model.
  • GPT-4-1106 is used as a uniform judge even when GPT-4 also generates the evaluated conversation.
  • In 50 dialogues with labels from two experts, the accuracy of the GPT-4 judge is 0.78 for openness, 0.90 for conscientiousness, 0.92 for extraversion, 0.80 for agreeableness, and 0.92 for neuroticism.
  • In BF-TC classification, GPT-4-1106 obtains mean F1 0.727; GPT-3.5-1106, 0.562; Zephyr 3-shot, 0.533; Vicuna, 0.528; and Zephyr zero-shot, 0.515.
  • The 3-shot of Zephyr does not improve generally: it raises especially agreeableness and neuroticism, but lowers openness and extraversion.
  • GPT-4-1106 reaches F1 0.741 on the binary linguistic ability label; GPT-3.5 remains at 0.660, Vicuna at 0.631, and Zephyr at 0.542.
  • The examples show that GPT-4 expresses low extraversion through silences, fillers, and hesitation, and high extraversion through more extensive and enthusiastic responses.
  • An example from the appendix assigns low agreeableness but is classified as high, a discrepancy consistent with the imperfect F1 of the system.
  • Without specifying BF-TC, the authors observe a default bias toward high poles of all traits except neuroticism.
  • The 500 generated BFI responses produce Cronbach's alpha from 0.906 to 0.936, but they are obtained from instructed simulated persons and not from a human population.
  • BFI correlations are positive among openness, conscientiousness, extraversion, and agreeableness and negative between neuroticism and the other four dimensions.
  • The mean BF-TC-BFI agreement varies between F1 0.725 for Zephyr and 0.802 for GPT-4; the two representations come from the same dialogue and the same simulated agent.
  • The higher ability condition is associated with more feedback, instructions, and questions; the lower ability condition, with more hints, explanations, and modeling.
  • The associations between BF-TC and scaffolding are visually more marked in the lower ability condition.
  • The article maintains that GPT-4 adjusts scaffolding to the profile, but that teacher indirectly knows the prescribed responses of the student within an entirely generated conversation.
  • The article acknowledges limitation to English and open risks of hallucination and bias.
  • No official publication of code, executable prompts, seeds, images, dialogues, labels, or complete tabular results is identified.

Limitations

  • The study evaluates obedience to role descriptions, not the psychological personality of an LLM or of a person.
  • BF-TC is defined with the same linguistic and behavioral signals that the classifier then seeks, which introduces circularity by construction.
  • Openness is mixed with creativity, acceptance of ideas, and interest in learning; conscientiousness with organization, attitude, and strategies; extraversion with fluency; agreeableness with interest and politeness; and neuroticism with confidence and sadness.
  • These reformulations confuse traits with ability, motivation, engagement, obedience, affect, and social desirability.
  • The low poles are described in normatively negative terms such as disinterest, lack of care, discourtesy, disorganization, and distraction, reinforcing stereotypes about introverted, anxious, or less agreeable students.
  • The attribution of silences, fillers, and hesitation to low extraversion may confuse personality with linguistic proficiency, anxiety, disability, culture, or communicative style.
  • The text says it draws inspiration from CHILDES, but does not document sampling, analysis, examples, annotations, or a reproducible empirical derivation from that corpus.
  • NAP is used as inspiration for two extreme prompts; the standardized protocol is not administered or validated in students.
  • The generated profiles are not compared with authentic conversations of children with independent measures of personality and ability.
  • There are no human participants who receive the tutoring, so there are no learning, engagement, satisfaction, safety, or retention outcomes.
  • The same model plays the teacher and the student, creating stylistic and knowledge dependence between the two roles.
  • GPT-4 judges all models and may judge its own generations; there is no independent evaluator for the complete set.
  • The human check covers only 50 dialogues and does not report who the experts are, training, blinding, labels per expert, inter-annotator agreement, adjudication, or distribution by condition.
  • The human-model judge accuracies do not include counts, confusion matrix, intervals, or comparison with baselines.
  • The article does not specify whether the labels of the two experts were combined by consensus, majority, or against an individual reference.
  • The composition of the 500 dialogues by model, image, linguistic level, and combination of traits is not fully explained.
  • The 100 images are described as open-sourced without identifying sources, licenses, content, or partition.
  • Temperature, top-p, seed, token limits, retry policy, or handling of invalid outputs are not published.
  • The open versions do not include exact identifiers of checkpoints, revisions, chat templates, or library versions.
  • GPT-3.5-1106 and GPT-4-1106 are closed services and versioned by provider; the outputs are not frozen in a public artifact.
  • The few-shot condition is tested only on Zephyr and does not include alternative examples, seeds, or an equivalent length control.
  • It is reported that GPT-4 significantly outperforms other models without a statistical test on the F1 of Table 2.
  • The F1 per trait have no intervals, variability across images, or systematic error analysis.
  • The embedding visualization does not specify embedding model, preprocessing, dimensionality reduction, hyperparameters, or a quantitative separation metric.
  • Answering the BFI after having received explicit trait descriptions favors prompt compliance and contaminates a supposed independent measure.
  • The BF-TC-BFI consistency uses as reference another classification derived from the same conversation and the same judge, not an external criterion.
  • A high Cronbach's alpha on conditioned synthetic responses does not establish construct, criterion, convergent, or predictive validity in people.
  • The exact criterion for converting BFI scores into high/low is ambiguously described as the mean and is not psychometrically justified.
  • The correlations among BFI scales reflect a mix of prompts, generated responses, and scoring; they do not test the factorial structure of the instrument.
  • No confirmatory factor analysis, invariance, test-retest, or comparison with human norms is performed.
  • The multiple correlations of traits and seven scaffolding types do not document complete correction for multiplicity, intervals, or independent effective sample size.
  • Turns within the same dialogue and dialogues derived from the same image are not necessarily independent observations.
  • The prescribed binary variables make correlations with teacher actions that respond directly to the generated text expected.
  • Scaffolding is not compared with decisions of human teachers, nor is it evaluated whether the adaptation is pedagogically appropriate or beneficial.
  • More feedback, hints, or questions does not by itself equate to better teaching or safe personalization.
  • The ethics section declares that it does not foresee unethical uses, but does not analyze profiling of minors, stigmatization, cultural bias, privacy, dependence, or automated educational decisions.
  • Only English and an image description task are addressed, with four models from 2023; generalization to other languages, ages, subjects, and models remains open.
  • Without code, generated corpus, and labels it is not possible to recalculate tables, figures, correlations, or classification errors.

What the study does not establish

  • It does not demonstrate that LLMs possess an internal or stable Big Five personality.
  • It does not demonstrate that BF-TC is a psychometrically valid adaptation of the Big Five.
  • It does not demonstrate that interest in learning, politeness, fluency, or confidence are valid and non-biased proxies of personality.
  • It does not demonstrate that the simulated profiles represent the behavior of real students.
  • It does not demonstrate that the BFI retains validity when an LLM knows the expected profile through a prompt.
  • It does not demonstrate that high Cronbach's alpha equates to construct validity or realism.
  • It does not demonstrate that BF-TC-BFI agreement is independent of the shared prompt and judge.
  • It does not demonstrate that GPT-4 is a reliable judge beyond the 50 checked dialogues or in current versions.
  • It does not demonstrate statistical superiority of GPT-4 over the other models.
  • It does not demonstrate that the scaffolding associations are causal.
  • It does not demonstrate that the labeled scaffolding is correct, personalized, safe, or effective.
  • It does not demonstrate improvement in learning, motivation, engagement, or well-being.
  • It does not demonstrate operation with real teachers, students, or images in a deployed ITS.
  • It does not demonstrate generalization to other traits, continuous levels, languages, cultures, ages, tasks, or models.
  • It does not allow auditing biases against introversion, neuroticism, low agreeableness, disability, or linguistic varieties.
  • It does not allow reproducing the 500 dialogues or the results without the missing experimental artifacts.

Traceability

Scope: Full text

Version: EMNLP 2024 main-conference final proceedings paper, pp. 626-642, DOI 10.18653/v1/2024.emnlp-main.37, 17 pages; no official code or data release identified in the paper, ACL record, or targeted project search

Consulted source: https://aclanthology.org/2024.emnlp-main.37.pdf

Review: Codex full-text, bilingual-fidelity, visual, bibliographic, educational-measurement, psychometric-circularity, LLM-judge, experimental-design, reproducibility, bias and ethics audit, 2026-07-15

Approval: Codex fidelity pass, 2026-07-15

English translation: approved, 2026-07-18

Models evaluated

  • Zephyr-7B-beta
  • Zephyr-7B-beta con generación 3-shot
  • Vicuna-13B-v1.5
  • GPT-3.5-1106
  • GPT-4-1106
  • PyTorch y Hugging Face Transformers para los modelos abiertos
  • OpenAI API para GPT-3.5 y GPT-4

Instruments and metrics

  • Big Five for Tutoring Conversation (BF-TC), construido por los autores
  • Big Five Inventory (BFI), 44 ítems y escala Likert de 1 a 5
  • Narrative Assessment Protocol (NAP) como inspiración para cinco componentes lingüísticos
  • Clasificación binaria de capacidad lingüística alta o baja
  • Rúbrica de siete estrategias de andamiaje: feedback, pistas, instrucción, explicación, modelado, preguntas y apoyo socioemocional
  • Etiquetado GPT-4 de diálogo y de enunciado
  • Precisión, recall y F1
  • Alfa de Cronbach y correlación de Pearson
  • Visualización no especificada de embeddings de respuestas

Data used

  • 100 imágenes de dibujos animados descritas como open-sourced, sin colección, licencia ni identificadores publicados
  • 500 conversaciones de tutoría generadas, según el artículo
  • 10.000 enunciados generados, según el artículo
  • 50 diálogos seleccionados aleatoriamente y etiquetados por dos expertos para contrastar al juez GPT-4
  • 500 respuestas sintéticas al BFI usadas en el análisis psicométrico
  • No se usa ni publica un conjunto de datos de estudiantes o docentes reales

Evidence and location

  • Bibliographic record and official abstract: ACL Anthology 2024.emnlp-main.37: 4 authors, EMNLP 2024, pp. 626-642, DOI 10.18653/v1/2024.emnlp-main.37
  • Full audited source: .cache/editorial-sources/article-097/source.pdf; official ACL PDF; 17 pages; sha256 8d6a5da1a2b82239c27dc82a9960b9b891eeaa664c5994e01507c38b742d61b0
  • High and low cognitive definition: Full text pp. 628-629, section 3.1
  • BF-TC definition and assumptions: Full text pp. 629-630, section 3.2 and Table 1; appendix pp. 640-641, Tables 6-8
  • Teacher and student prompts: Full text p. 630, Codeboxes C1-C2; appendix p. 639, validation instructions
  • Validation of traits, ability, BFI, and scaffolding: Full text p. 630, section 3.3; appendix pp. 639 and 642, sections A.1-A.2 and Table 9
  • Four models, roles, and generated corpus: Full text p. 631, sections 4.1-4.2: same model for teacher and student, 100 images, 500 dialogues, 10K utterances
  • Human check of the GPT-4 judge: Full text p. 632, section 5.1: 50 randomly selected dialogues, two experts, five reported accuracies
  • BF-TC and linguistic ability results: Full text p. 631, Tables 2-3
  • BFI results and BF-TC-BFI consistency: Full text pp. 632-633, Tables 4-5 and section 5.2
  • Default bias without BF-TC: Full text p. 632, footnote 1
  • Embeddings without reproducible specification: Full text p. 632, Figure 3; no embedding model or dimensionality-reduction method is stated elsewhere in the paper
  • Associations with scaffolding: Full text pp. 633-634, Figures 4-5 and section 5.3
  • Examples and agreeableness discrepancy: Appendix pp. 641-642, Tables 8-9
  • Computational environment: Appendix p. 639, section A.2: single A100 80G, PyTorch, Hugging Face Transformers and OpenAI API
  • Limitations and ethics statement of authors: Full text p. 634, Limitations and Ethics and Impact Statement
  • Absence of reproducible artifact: Paper and ACL metadata contain no project, code or data URL; targeted official GitHub/project search checked 15 Jul 2026
  • Integral visual check: All 17 PDF pages rendered and visually inspected, including five result tables, five figures, prompts, validation instructions, dialogue examples, limitations and ethics; checked 15 Jul 2026