Roleplay-doh is a tool for a mental-health-support expert to construct a simulated patient from a remembered case, converse with it, and convert kudos, critiques or rewritten responses into behavioral principles. GPT-3.5 turns kudos and critiques into rules; GPT-4 explains a rewrite and derives the rule. For every dialogue turn, gpt-4-turbo-1106 receives the scenario, principles and history. A second pipeline decomposes principles into yes/no questions, adds general dialogue criteria, decides which criteria apply, evaluates the initial response and rewrites it if any applicable criterion fails. The pilot involved six 7 Cups counselors; four coauthors then conversed with four selected patients and rated 276 responses, with only one rater per response. Fifty-five, or 20%, received moderate or low satisfaction, but this estimate has no inter-rater agreement or independent expert adjudication. The main within-subject study included 25 US professionals or counselors. Everyone first held a ten-minute Scenario-Only conversation and then a thirty-minute session about the same remembered case while adding rules. They created 25 patient pairs and 123 principles. Creators rated the principle version higher on five of six scales: increases were 0.80 for authenticity, 0.76 for resemblance to the remembered case, 1.00 for challenging aspects, 0.64 for training readiness and 0.52 for recommendation; staying in role barely changed, by 0.08. The design was not counterbalanced, however: order, practice, session length and tool use changed together, while creators knew the condition and judged their own rules. Five counselors drawn from the same creator cohort rated randomized, blinded transcripts for all 25 pairs, producing 125 comparisons. A mixed model, Rating~Treatment+CreatorID+(1|AnnotatorID), estimated smaller increases for authenticity (+0.31), typical-case resemblance (+0.49), readiness (+0.39) and recommendation (+0.38); role (+0.09) and counselor challenge (+0.22) were not significant. Agreement was poor: Krippendorff alpha was 0.023-0.082 for rating differences and 0.046-0.232 for preference. The technical evaluation compared five variants with three expert counselors. On 40 selected failures, Full won, tied and lost against the initial response in 30%, 55% and 15% of overall judgments. That 30% is a win proportion, not a 30% improvement on a quality scale. On 50 random turns the result was 18%/78%/4%, and in 34 of 50 turns every method returned exactly the same response. The work therefore supports that principles are authorable and that this small sample perceived modest improvements in some transcripts; it does not establish novice learning, clinical validity, safety or transfer to real patients. Scenarios came from counselors' memories of real patients, but the paper does not document patient consent, de-identification or a privacy review. Although the project page promises code and data, its only official repository contains the static website and the Code button is commented out. The principle dataset is on Hugging Face under CC BY-NC 4.0 but requires authentication and contact sharing; the application, analysis, raw study data and model outputs needed for end-to-end reproduction are not public.
Research question
Can a tool convert expert feedback into principles that govern LLM-simulated patients, improve adherence to those rules, and produce transcripts that counselors perceive as more authentic and useful than a scenario without principles?