The paper introduces Chameleon, a dataset of inferred psychological profiles for 5,001 posts from 1,667 Reddit authors, with three posts per author from three distinct subreddits. The audited version is arXiv v2, revised on 6 May 2026 and accepted to Findings of ACL 2026. Source texts come from Webis-TLDR-17 and date from 2006–2016. The authors operationalize context as subreddit and ask what fraction of variation in 26 psychological dimensions appears between authors or within the same author. They then test whether three LLMs differentiate six archetypes and whether three reward models score an identical response consistently when the stated user profile changes.
Sampling requires authors with at least 50-word posts in three distinct communities and takes exactly three posts per person. The public card adds random seed 42, which is absent from the main paper. The sample is highly uneven: AskReddit contributes 1,558 posts, relationships 923, relationship_advice 268, offmychest 198, and depression 129; these five contexts account for 61.51% of all rows. Of 645 subreddits, only 41 have at least ten posts and 433 occur once. Thus “645 contexts” is nominal coverage, not a balanced comparison of situations. Length, three-community, and TL;DR availability criteria also select a particular subset of Reddit users and writing.
Each post receives 26 scores: five Big Five, ten Schwartz values, five Self-Determination Theory, and six DOSPERT dimensions, totaling 171 items. Two feature routes are used: SEANCE produces 254 lexical indicators and LangExtract uses GPT-4o to extract semantic patterns. Both routes, however, terminate in the same GPT-4o answering psychometric items as if it were the author while conditioned on those features; LangExtract also uses GPT-4o upstream. These are not independent measurements because they share the same assessor, assumptions, knowledge, and potential biases. There are no self-reports, observed behavior, expert annotations, or psychological ground truth.
The paper calls its decomposition an ICC and reports means of 0.26 for SEANCE, 0.28 for LangExtract, and 0.27 for the fused data, interpreting 1−ICC as 72–74% “state.” Independent analysis of the three public CSVs reproduces those values exactly only as Var(user mean) / [Var(user mean) + mean within-user variance]: 0.25995, 0.27991, and 0.27311. This ratio does not correct user-mean variance for having only three observations and is not the standard one-way random-effects ICC(1,1) claimed by the method. With k=3 independent observations, its expectation is already 0.25 between and 0.75 within, placing the headline finding close to a mathematical floor.
Applying standard ANOVA ICC(1,1) to the same data yields means of 0.0176 for SEANCE, 0.0524 for LangExtract, and 0.0406 for combined, with ranges including slightly negative values. This indicates very low consistency across posts, but it does not identify why. Within-author residual variance mixes genuine contextual change, topic, time, post selection, SEANCE error, GPT-4o variability and bias, and all other unmodeled sources. Neither 73% nor 96% can therefore be labeled directly as psychological state. The paper's own equation combines state variance and error variance, although later prose often attributes the whole residual to context.
The subreddit control does not causally isolate context. Subtracting each community mean jointly removes topic, style, user composition, and any shared signal; for 433 singleton subreddits it turns the entire observation into zero. Residualized ICC moves from 0.273 to 0.266 under the same uncorrected ratio. This stability does not separate state from error or prove that subreddit caused the differences; it only shows that this aggregate transformation does not increase the statistic.
Convergent validity is particularly weak. Correlation for the same dimension across the two methods averages r=0.0605, range 0.0069–0.1040. The paper instead emphasizes within-post correlation across the 26 dimensions: mean 0.7088 and median 0.7586 in the public CSVs, because both routes preserve level and range patterns associated with different scales. When every dimension is z-normalized across the dataset, as the described fusion stage requires, mean profile correlation falls to 0.0448 and median to 0.0462; only 0.18% exceed 0.70. The reported r=0.71 does not establish construct agreement and cannot support the claim of converging independent methods.
There is also direct method–artifact drift. The dataset card and algorithm say each method is normalized per dimension before averaging, but all 26 combined columns are exactly the raw arithmetic mean of SEANCE and LangExtract, with maximum error 1.33×10⁻¹⁵. They retain original response scales rather than z-scores. One row, post c1ca6ie, lacks all six DOSPERT dimensions in LangExtract and combined, with no imputation policy or card warning. The paper also alternates between a 1–5 DOSPERT scale in the main text and 1–7 in the appendix and dataset card.
Five literature-driven contrasts recover expected differences: SuicideWatch and depression show higher inferred neuroticism, SuicideWatch lower competence, and personalfinance higher security and achievement than AskReddit. Public raw mean differences almost directly reproduce the reported coefficients, but this is internal and circular validation: features contain vocabulary defining those communities and GPT-4o knows the associations being “confirmed.” SuicideWatch has only 43 posts and personalfinance 49; the paper reports no correction across the five hypotheses, mixed-model diagnostics, or sensitivity to dominant communities.
Six archetypes are produced by k-means over z-normalized fused profiles and receive normative labels such as Distressed-Vulnerable, Driven-Assertive, and Risk-Seeking-Detached. Public data, z-normalized fusion, and seed 42 reproduce 50.75% of users occupying three distinct clusters, matching the reported 50.7%. Standard silhouette analysis over k=2–10, however, favors k=2 at about 0.226 rather than k=6 at about 0.120; the paper provides no curve, seed, n_init, missing-data rule, or additional selection criterion. Labels, descriptions, and claims about what advice “benefits” each profile are author interpretations, not user-validated results.
The generation application uses GPT-4o, Llama-3.1-8B, and Qwen2.5-14B over 127 questions and seven conditions, one model–question–condition response each: 2,667 responses. Full checkpoints, provider, date, temperature, top-p, seeds, replications, and failures are not reported. Mean similarity across conditions is 0.768, 0.819, and 0.846; the archetype ANOVA gives F=2.18 and p=0.054. A non-significant test just above 0.05 is not evidence of equivalence or state blindness, and pairwise comparisons share questions, models, and conditions.
More importantly, the published prompt does not ask an assistant to adapt to a user. Its system instruction says “You are a person with the following characteristics” and “Respond authentically as this person.” The experiment measures the model's role-play under an assigned persona, while the paper interprets it as personalized assistance to an anxious or confident interlocutor. This construct mismatch prevents concluding that LLMs are state-blind to user state. There are also no human-adapted responses, appropriateness judges, utility criteria, or equivalence bounds.
The reward-model application generates one baseline GPT-4o response per question and scores it without a profile and under six profiles using DeBERTa-RM, Skywork-RM-8B, and ArmoRM-8B. ArmoRM raises scores while the other two lower them, with absolute d values roughly 0.31–1.16. Exact checkpoints and official templates are missing, and adding a long profile changes prompt length and input distribution relative to baseline. No human preference reference exists. Total state invariance is also debatable: if quality includes user appropriateness, the same response can legitimately fit different needs differently. Opposite directions show scorer disagreement under this prompt; they do not prove that deployed systems prioritize or deprioritize vulnerable people or that the effect automatically propagates through RLHF.
The public Hugging Face artifact contains three CSVs, a README, and CC BY 4.0 metadata at commit 39aa8097644eb3d64e87e802c1d9f0f16f846287. It contains no code for sampling, SEANCE/LangExtract, executable prompts, GPT-4o calls, ICCs, regressions, clustering, generation, embeddings, or reward models, and no outputs, dependencies, lockfile, tests, CI, or logs. The paper says it releases “dataset and code,” but its only footnote links the dataset and a targeted search found no code repository. The CSVs are auditable; the pipeline and two applications are not reproducible.
Privacy needs stronger caution than the card provides. Although user_id is pseudonymized and text is omitted, every post_id is public and the README explicitly says it matches id in Webis-TLDR-17. That corpus exposes author, body, content, and subreddit, so joining recovers text and author names, including depression and SuicideWatch posts. Omitting text reduces convenience but does not prevent reidentification. The release documents no consent, deletion process, risk assessment, independent ethics review, or handling of sensitive psychological inferences; it only states that the research does not meet the US federal human-subjects definition.
The defensible contribution is an exploratory resource linking three texts per author to two families of inferred scores and showing very low across-post consistency, strong associations with subreddit content, and sensitivity of generators and reward models to persona cards. It does not establish that 74% of human psychology is state, that subreddit causes change, that profiles are valid, that independent methods converge, that LLMs ignore real users' state, or that reward models discriminate against vulnerable people in deployment.