Yan and colleagues ask whether an LLM can estimate Big Five traits from Chinese counselling dialogues. Their procedure does not directly predict five numbers: it places each session into a client, counsellor, or observer role, administers all 60 BFI-2 items to the model, extracts Likert choices with a regex, and computes OCEAN using the inventory scoring rules. The criterion is the BFI-2 completed by each client before their first session. The corpus contains 853 sessions and 65,347 utterances from 82 adults and nine counsellors, although the abstract and introduction say 83 clients. The authors allocate 611 sessions to training and 242 to validation but do not say that the split is client-disjoint, even though each person contributes roughly ten sessions sharing one trait label. In zero-shot evaluation, combining the client role with the questionnaire greatly improves over direct prediction: Qwen1.5-110B rises from mean PCC 0.172 to 0.426 and DeepSeek-Chat from 0.080 to 0.395. The questionnaire without a role already accounts for most of the gain (0.319 and 0.284); role-play alone does not help. Across 21 models, Qwen1.5-110B and Gemini-1.5-Pro reach mean 0.425, while several smaller models are near zero. Roles closer to the dialogue perform better on average: client exceeds counsellor, observer, and no-role, though not for every trait. With 30% of the dialogue, correlations are already significant for the tuned model (mean PCC 0.510), but they continue to rise to 0.631 at 90%; Qwen obtains 0.331 at 30% versus 0.425 with the full context. Thus, 30% is a significance threshold, not evidence of equivalent or practically sufficient performance. To align Llama-3-8B, the authors choose generated responses with the lowest error as preferred and the highest-error responses as rejected, then apply DPO with an SFT constraint. Llama-3-8b-BFI reaches PCC 0.692, 0.554, 0.569, 0.448, and 0.648 for O, C, E, A, and N, averaging 0.582. The reported relative gains of 130.95% over its base (0.252) and 36.94% over Qwen (0.425) are arithmetically correct, while the absolute gains are 0.330 and 0.157. The full matrix contains large cross-trait correlations, for example, predicted O correlates 0.522 with actual E and -0.455 with actual A, limiting discriminant validity. Error analysis provides plausible examples of extracting states and also of misunderstanding, stereotyping, and safety refusal; calling this “content validity” does not replace expert psychometric assessment. Reliability is also ambiguous: the paper reports one alpha per model, trait-level kappas in one table, and PCC across ten runs in another without sufficiently defining units and calculations. Consent, compensation, ethics approval, and offline-only use are documented. The evidence supports that decomposing the task into BFI responses and tuning on error-labelled preferences yields moderate correlations within this corpus; it does not establish an unbiased or safe automated assessment for clinical use.
Research question
Can LLMs estimate OCEAN from real psychological counseling sessions in Chinese, what role do role-play, BFI decomposition, amount of context, and model capacity play, and how much does a DPO plus SFT alignment improve?