The paper adapts the psychological logic of informant reports to LLM agents. Instead of asking only a subject model to complete a questionnaire about itself, multiple observer agents converse with the subject in friendship, family, or workplace scenarios and then rate its behavior using 50 IPIP Big Five items. The main experiment creates 100 prompt-conditioned subjects, 15 observers per subject, five for each relational context, and five scenarios per subject–observer pair. GPT-4o generates relationships and scenarios and fills both subject and observer roles; the appendix repeats analyses with Qwen2.5-72B-Instruct and Llama-3-70B-Instruct. Ratings from groups of 5 to 15 observers are averaged and compared with the injected profile, self-report, and a small human evaluation.
The central result is a separation between instruction and simulated behavior. Self-report correlates almost perfectly with the trait profile inserted into the prompt, 0.93 to 0.97 depending on the dimension, whereas aggregated observer reports correlate less with that instruction, 0.55 to 0.86, but generally approximate human judgments of the dialogues more closely. In the main table, observer reports outperform self-report against human judgment for openness, agreeableness, and neuroticism and are slightly lower for conscientiousness and extraversion; in the appendix, their absolute distance is smaller for all five dimensions and all three models. Correlation curves stabilize at roughly five to seven observers. Systematic deviations also appear: observers rate agreeableness 0.91 points and conscientiousness 0.39 points above the subject’s self-rating, with Cohen’s d values of 1.07 and 0.46; family, friendship, and workplace contexts affect these two dimensions.
The valuable contribution is not evidence of internal personality but evidence that a declared profile can diverge from generated behavior. The qualitative case is illustrative: a subject instructed to be highly disagreeable gives itself an agreeableness score of 1.7, but produces a cautious and fairly cooperative dialogue and receives observer ratings around 3. The design supplements self-report with behavioral material, tests three model families, four prompt variants, reversed scale order and batch presentation, and adds a human comparison with inter-rater agreement. It is a promising methodological direction for measuring behavioral fidelity in induced personas.
However, the “informants” are not independent external observers. In the main condition, they are instances of the same GPT-4o that generates profiles, relationships, scenarios, both sides of the dialogue, and IPIP ratings. Observers and humans see the dialogues, while self-report directly receives the personality markers; this asymmetry structurally favors observer–human agreement and self-report–prompt agreement. The injected profile is not psychological ground truth, averaging correlated agents does not establish reliability, and the five-to-seven plateau is not validated with confidence intervals, resampling, or an external test set. Human evidence covers 16 cases with two raters per case, while the Responsible NLP checklist instead mentions 12 participants and confirms that no ethics-board approval or exemption was obtained. The repository declared available in the final paper returns 404, so prompts, outputs, and analyses are not currently reproducible. The defensible conclusion is that multi-agent reports better capture human impressions of these synthetic conversations under this protocol, not that they measure a true or more objective LLM personality.