PACIFIC is a synthetic multiple-choice benchmark for testing whether organizing preferences by Big Five labels helps an LLM choose a congruent option; it is neither a collection of observed human preferences nor a psychometric test. Gemini 2.5 Pro generates 1,200 preference-question-four-choice triples across 20 topics and ten trait directions, high/low O, C, E, A, and N. Another LLM, GPT-4o-mini, scores every text from 1 to 7 per trait and assigns confidence. A .7 filter leaves 803 candidate preferences; the main experiments use 200 balanced questions, 20 per direction.
For Gemma-3-4B-IT, Table 1 reports 25.75% without preferences, 29.25% with five mixed preferences, 63% with five preferences from the correct trait, and 61.75% after adding two distractors. Adding ground-truth labels to the preferences reaches 76%, while an unlabeled reminder reaches 67%. Labeling choices as well drops to 57.25%, and labels without text to 37.75%, with collapse on several low traits. Llama-3-8B-Instruct reaches 88.75% with aligned preferences; Gemini-2.5-Pro reaches 99.25% and GPT-4o-mini 97.5%, near ceiling on a corpus generated to expose the same rule.
The 29.25→76% headline combines two interventions: ground-truth selection of aligned raw preferences raises Gemma to 63%, then adding oracle labels raises it to 76%. This is not the automatic system's result. Without labels, pretrained DPR scores 30.25% and the contrastively fine-tuned retriever 43%. End-to-end personality inference is also weak: Gemma reaches 54.5% from preferences, with 95–100% on high traits but only 0–40% on low traits; prediction from choices reaches 85.69%. The paper attributes this asymmetry to RLHF-induced social-desirability bias, but it does not compare a base and RLHF model or eliminate prompt and dataset artifacts, so the causal attribution remains a hypothesis.
The human check has 15 participants complete an unnamed personality assessment and evaluate 25 cases routed to their dominant traits. It reports 78.22% accuracy, Fleiss' κ=.8599, Cohen's κ=.9170 between GPT-4o-mini and consensus, and 67.11% personal resonance. The instrument, allocation of people and items, full instructions, recruitment, compensation, uncertainty, mismatched-trait control, and ethics review are absent. A 67.11% approval rate is not a psychometric correlation.
The official Hugging Face artifact at commit dee6c7144ca0efa1132eebfdd538ecbde610b927 has 1,200 rows in one train split: exactly 120 per trait direction and 60 per topic, despite the paper's claim that it does not impose uniform distributions. Every row has ground_truth_choice_id=0 because the generation prompt requires the correct response first. No shuffling is documented; the non-100% reported scores therefore require an undisclosed preprocessing step or a different protocol that cannot be checked. The release also lacks the 2–4-turn conversations the paper says it adds; its card says they will come later. No code, 200-item sample IDs, retriever train/evaluation split, model outputs, or runnable results are public. PACIFIC is useful for showing how constructed OCEAN labels can organize context, but its evidence measures consistency with a synthetic taxonomy rather than real people's preferences.