LMLPA proposes replacing direct administration of the Big Five Inventory to an LLM with a questionnaire adapted to linguistic capabilities. It converts the 44 BFI items into open-ended “to what extent” questions, requires the model to include always, often, sometimes, rarely, or never plus a one-sentence rationale, and asks another model to turn the text into a 1–5 score. Five psychology professionals qualitatively reviewed the adaptations; PCA then removes four items, leaving a final 40-item version. Its clearest contribution is the complete publication of questions, prompts, labels, and order-sensitivity analyses, while explicitly acknowledging that LLMs do not have human actions, emotions, or cognitive processes and that output is used as a linguistic proxy.
In the order tests, GPT-4-Turbo answers all 44 items in separate calls at temperature 0. Reversing the required adverb list yields seven pairs considered truly inconsistent after one semantically equivalent case is manually corrected, with weighted kappa 0.730. In the later comparison, numeric BFI produces 16 discrepancies and kappa 0.401, whereas the full LMLPA system produces six and kappa 0.877. This demonstrates greater output stability under those two configurations, but it does not isolate the effect of an open questionnaire: the BFI condition reverses the scale shown to the test-taking model, while LMLPA reverses the scale shown to GPT-4-Turbo as judge. An ordered list of five adverbs remains in the test-taker prompt, and a list of five labels is shifted to the rater, so option structure is moved to another stage rather than removed.
Three of the experts rate one set of 44 GPT-4-Turbo responses. GPT-4-Turbo as rater correlates 0.827–0.877 with them and has a single-measure ICC of 0.829; BART-Large-MNLI reaches 0.700–0.810 and ICC 0.766, and Llama3-8B reaches 0.750–0.840 and ICC 0.785. This is evidence of agreement on highly structured responses containing a mandatory frequency adverb, not evidence that the inferred trait is correct. The exact ICC model and whether it targets agreement or consistency are not specified; the “average measures” ICC combines four ratings and is not the performance of one automated judge. Llama3 also obtains kappa 0.279 when its rating options are reversed, so it is excluded from the precise psychometric phase.
Reliability and structure are estimated from 250 conditions generated by the same GPT-4-Turbo: ten PersonaChat descriptions crossed with 25 profiles that explicitly place five levels of each trait in the system prompt. Alpha values on the 44 items are 0.869–0.936, but can reflect item redundancy, prompt compliance, and shared test-taker/rater method; they are not recomputed after Q31, Q35, Q41, and Q22 are removed to form the final 40-item scale. PCA reports KMO 0.951 and extracts four components rather than five; Conscientiousness is distributed across several components, cross-loadings are extensive, and the prose assigns Neuroticism to component 3 even though the table concentrates it in component 2. The same sample is used for item deletion and the validity claim, with no CFA or independent replication.
The final test again treats Big Five adjectives written directly into prompts for GPT-4-Turbo, Llama3-8B-Instruct, and Mistral-7B-Instruct-v0.2 as “ground truth.” Score distributions usually shift in the requested direction but do not match target levels; low Agreeableness clusters at 2–3, and Llama3 Neuroticism moves in the opposite direction at the extreme. No error, correlation, or classification metric is reported. When GPT-4-Turbo both generates and judges outputs, model circularity and shared instrument knowledge remain. Without exact service snapshots, seeds, repetitions, outputs, data, or public code, the full protocol is not reproducible. LMLPA is therefore a promising open instrument for describing prompt-conditioned linguistic self-reports, but its evidence does not establish a latent, stable, predictive, or human-equivalent Big Five personality.