This CLPsych 2025 paper does not study personality or theory of mind inside LLMs; it uses models as binary graders of human open-ended answers to ten social narratives intended to measure advanced mind-reading. A total of 1,733 people aged 13-30 participated through schools or Prolific. One lead coder and four trained coders assigned labels; before independent coding they reached kappa greater than .7 against the lead, who resolved discrepancies. Each prompt includes the narrative, question, detailed rubric, and answer, and the model returns 0/1. The paper compares plain/XML/JSON, three rubric versions, zero-shot, 10-shot, 50-shot, and fine-tuning. It reports 89.4% macro accuracy for zero-shot GPT-4o and 92.8% after fine-tuning, with kappa .83. RoBERTa-large reaches 92.0%, GPT-4o-mini 90.5%, Llama-3.2-3B 91.1%, Mistral 88.7%, Phi-3.5 83.8%, Phi-4 87.5%, and Longformer 86.7%. GPT-4o paraphrase augmentation helps Longformer, Mistral, and Phi-3.5, barely changes Phi-4, and slightly harms Llama. These are promising fixed-rubric imitation results, but artifact audit blocks the paper's broader interpretation. Code creates 17,330 response opportunities, randomly removes 3,078 majority-class rows before splitting, and publishes 14,252 balanced rows. The actual split is 11,543/1,283/1,426-80.99%/9.00%/10.01%, not 80/10/10. The pipeline also drops participant IDs before `train_test_split`; because each person answers all ten narratives, participant-specific style can occur in train and test. All ten items and rubrics appear in every split, so the experiment tests new answers to known items rather than transfer to new items or instruments. Three exact prompts overlap train/validation and one overlaps train/test. Although test is globally 50/50, its per-item majority baselines average 62.03% and range from 50.32% to 77.70%; zero-shot Llama at 64.3% is only 2.27 points higher. The paper calls several differences significant without tests, intervals, replications, or participant/item bootstrap. Its human comparison is more problematic: kappa is computed only on cases where GPT-4o and the coder disagreed, conditioning the sample, with the lead coder as reference. The prose says GPT-4o wins except on item 5, but Table 10 also has the human ahead on item 3 and contains no item 7 despite listing it in the text. The repository also cannot reproduce the headline: it lacks the OpenAI pipeline, predictions, results, few-shot/format and BERT/RoBERTa code; the paraphrase dataset returns 401, `requirements.txt` is missing, scripts contain broken paths, README swaps Phi/Mistral links, and checkpoints are incomplete. The defensible conclusion is that fine-tuned models can approximate a lead coder on ten known questions in a reduced balanced corpus. The work does not validate the test, generalization to new participants, items, demographic groups, or clinics, or use as a screening tool.
Research question
With what accuracy and agreement with a lead coder can various models score human open-ended responses on ten mind-reading items, and how do results change with format, examples, fine-tuning, and synthetic paraphrases?