This preprint compares ten graduate psychology students with five MPNet embedding models on recovery of the officially assigned Big Five factor for items from the Italian Big Five Questionnaire (BFQ) and the Big Five Inventory (BFI). For BFQ, 50 of 132 items are selected, ten per factor. The method introduces the 44-item BFI, but results, percentages, and plots use eight items per factor, 40 total, without explaining which four are excluded. Italian judges score each item against each construct as 0, 1, or 2. Although the paper calls this the Content Validity Ratio, final classification does not use the published CVR statistic or its critical threshold; it selects the construct receiving the most positive ratings. Models are paraphrase-multilingual-mpnet-base-v2 on Italian, all-mpnet-base-v2 on an English translation, SurveyBot3000, dwulff/mpnet-personality, and a new all-MPNet variant fine-tuned by the authors. For each item, the algorithm averages cosine similarity to all other items whose theoretical labels are already known, separately by factor, and selects the maximum after softmax. This is not unsupervised: it uses labels for every other benchmark item and is transductive leave-one-item-out classification. Softmax adds neither evidence nor calibrated probability; it preserves the same argmax. The useful result is descriptive. On BFQ the final table reports human 84%, multilingual MPNet 64%, English MPNet 70%, SurveyBot 64%, Personality MPNet 72%, and the authors’ model 80%. On BFI it reports human 72%, multilingual 77.5%, English 80%, SurveyBot 75%, Personality MPNet 97.5%, and the authors’ model 82.5%. Personality MPNet’s BFI advantage corresponds to recovering 39/40 labels versus 29/40 for humans; it does not establish content validity or superiority over test-construction experts. The task evaluates recovery of official labels, not whether an item set comprehensively and representatively covers a construct domain. Facet coverage, relevance, clarity, cultural bias, and construct-irrelevant variance are not measured. Language comparison is confounded: humans and multilingual MPNet see Italian, while English models see undocumented translations; no crossed design separates model, language, and translation. The new model is trained on IPIP pairs pseudo-labeled by the retrieval cross-encoder cross-encoder/ms-marco-MiniLM-L12-v2. The paper says 3,250 items yield more than 15 million pairs, but there are 5,279,625 unordered pairs and 10,559,250 ordered non-self pairs; with the 3,320 items mentioned elsewhere, counts would be 5,509,540 or 11,019,080. It also describes cross-encoder outputs as similarities from −1 to 1, while the official model card shows logits such as 9.219 and −4.078. No normalization, clipping, or loss details are given. The authors omit a train/test split because exact BFI and BFQ items are said not to occur in IPIP, but this does not evaluate fine-tuning generalization, rule out near-duplicate inventory items, or validate learning from millions of dependent pairs. Promised supplementary material is absent from the PDF and was not located elsewhere: the item list, Figures S1–S16, and training details are missing. Internal contradictions remain. Human BFQ is repeatedly described as 82%, while factor results and the final table sum to 42/50=84%. Human BFI sums to 29/40=72.5%, not 72%. Personality MPNet is said to raise BFQ Neuroticism from 50% to 100%, but base MPNet is already reported at 100%. The authors’ model is said to improve all BFQ constructs, although it ties base MPNet on Agreeableness and Neuroticism. “Significant” is used without a test, interval, or repeated sample; 80% versus 84% on BFQ is two items. The faithful conclusion is that some embeddings recover Big Five labels accurately on these two sets under a transductive procedure and that a model trained on inventory correlations reaches 39/40 on the BFI subset. The study does not automatically validate test content, demonstrate objectivity or robustness, or establish a hybrid advantage because no system combining human and model decisions is tested.
Research question
How accurately do ten judges and several MPNet embeddings recover the official Big Five label of BFQ/BFI items, how do language and fine-tuning influence this, and what are the limits of using semantic similarity as an aid to content validity?