InCharacter evaluates whether role-playing agents reproduce personality labels attributed to their target characters. Instead of asking an agent to complete a questionnaire directly, it rewrites items from 14 scales as open-ended questions, asks each in an isolated context, and uses another LLM either to convert the answer back to an option or to score a dimension as a simulated expert. A preliminary validation compares GPT-3.5, GPT-4, and Gemini with human judgments on 100 BFI cases. GPT-4 reaches 89% accuracy in expert rating and Pearson, Spearman, and Kendall correlations of 0.925, 0.927, and 0.837, while still differing by more than one point in four cases. The main benchmark covers 32 popular characters: 16 ChatHaruhi and 16 RoleLLM configurations, normally run on gpt-3.5-turbo-1106. For BFI and 16Personalities, positive or negative types come from Personality Database vote percentages, whereas continuous distances use scores from two or three invited annotators per character. With batched expert rating and GPT-4, Table 2 reports 76.6% dimension accuracy, 31.2% all-dimensions accuracy, and 18.2% normalized absolute error for BFI; corresponding 16Personalities results are 80.7%, 44.8%, and 20.5%. Thus, the abstract's 80.7% does not mean that 80.7% of characters have their entire personality reproduced correctly. Across 14 scales using GPT-3.5 as evaluator, published averages are 78.9% per dimension, 58.5% full accuracy, and 8.1% error, but mean inter-annotator reliability is κ = 0.609 and falls to 0.438 for EPQ-R, 0.473 for ECR-R, and 0.463 for CABIN. Interview methods generally outperform self-report on the selected metrics and yield more dispersed character measurements, although the comparison is asymmetric: refusals or non-compliant self-reports are mapped to neutral, while evaluator failures are regenerated and Gemini may be replaced by GPT-4. Name masking also leaves narrative clues that may reveal the character to the evaluator. The study contains no real-person assessment, longitudinal behavior, or test of whether the scales retain construct validity when applied to agents. The official repository publishes compilable code, 14 questionnaires, and aggregated labels under MIT, but it does not contain the announced 18,304 dialogues, result files, or individual annotations; dependencies are unpinned, and reproduction requires retired model snapshots, external services, and an unofficial 16Personalities API. The evidence supports partial agreement with human labels and discrimination among configurations within this benchmark; it does not establish internal personality or general character fidelity.
Research question
Can the personality of a role-playing agent be measured with greater fidelity through open-ended questions derived from psychological scales and subsequent evaluation by another LLM than through direct self-report, and to what extent do the resulting measurements coincide with human labels of the target characters?