The Findings of EMNLP 2025 paper introduces LIFECHOICE as a narrative decision-reconstruction benchmark: given a character, context preceding a decision, a scenario, and four options, a role-playing agent must select the action the character took in the novel. The paper reports 2,512 decision points from 470 books. It uses Supersummary character descriptions, chapter summaries, and book analyses as expert-written material. GPT-4 selects decisions, motivations, and relevant chapters and, while seeing the gold decision and motivation, constructs the scenario, question, correct option, and three distractors. Ten English-speaking university students filter items, and GPT-4o assigns one motivation category per example. Profile methods include GPT-3.5 recursive and progressive summaries, expert descriptions, BM25 or `text-embedding-ada-002` memories, direct concatenation, and CHARMAP. CHARMAP locates question-relevant episodes in a description and uses them to retrieve memories. In a single run per condition, the best reported accuracy is 65.28% for o1+CHARMAP versus 61.12% for o1+direct concatenation, a descriptive 4.16-point difference. The paper also reports 92.01% human accuracy with raw text, but this comes from three students and six unfamiliar novels under a small protocol with no exact question count, assignment design, agreement, or uncertainty; it is not a matched benchmark-wide ceiling. Audit of the official repository materially changes the interpretation. It releases only 1,576 records from 383 books, 62.74% of reported items and 81.49% of books, and omits original books, CHARMAP, profile construction, retrieval, outputs, analysis, and the exact environment. Its only script tests older or generic models unlike those in the paper, puts only the name under Description, treats `input_text` as Memory, duplicates option letters, and saves no predictions. The public subset has ten invalid rows and extreme position imbalance: B is correct 837 times (53.11%), C 618 (39.21%), A 48, and D 63. Always choosing B scores 53.11%, above or near many description-only and memory-only results and far from a uniform 25% chance assumption. Whether the full set has the same imbalance cannot be checked because it is unreleased. Direct semantic leakage also appears: the first scenario says “Upon refusal, Leduc threatens to expel her” and then asks what Amelia decides; the gold answer is to reject Leduc. Because GPT-4 saw the decision and motivation while writing questions and the paper lacks question-only, random-profile, shuffled-character, option-order, or answer-blind rewrite controls, accuracy can measure wording and generator cues in addition to persona understanding. Claims that CHARMAP “significantly advances” performance and model gaps are “insignificant” have no statistical tests, intervals, repeated runs, or variance. The contamination analysis uses Douban popularity and 30 post-cutoff books as proxies; it cannot establish absence of plots or Supersummary text from training. The defensible contribution is an interesting narrative role-playing benchmark and a promising retrieval method; the released evidence cannot reproduce its tables or establish personality understanding, human fidelity, real-world choice prediction, or statistically reliable superiority.
Research question
To what extent can LLMs and role-playing agents reconstruct the original decision of a fictional character from their preceding history, and does a specific scenario profile combining description and retrieved memory improve accuracy?