This paper presents a framework for studying uncertainty and fairness in LLM-generated recommendations and an experiment with an endpoint labelled Gemini 1.5 Flash. Its defensible empirical contribution is narrower than the abstract: song and movie lists from a neutral prompt are compared with lists from prompts containing demographic labels. This demonstrates sensitivity to textual persona wording, but it does not measure psychometric personality, recommendation quality, fairness harm, or original predictive uncertainty. The audit covered all 15 pages of arXiv v1, all 9 appendix pages, and the official code/data ZIP at commit 3c5ff5e.
The paper asks two questions. RQ1 concerns how predictive uncertainty, purportedly quantified with entropy, affects reliability. RQ2 asks whether observed Gemini disparities persist across sensitive attributes, domains, typos, and multilingual prompts. RQ1, however, contains no author-run experiment: it is a narrative synthesis of prior studies in Table 2. No entropy is defined or estimated over the paper's outputs; there is no repeated sampling, token probability, calibration result, empirical uncertainty table, or association between uncertainty and accuracy. The artifact contains no entropy or token-level uncertainty computation either. Claims that uncertainty degraded this experiment and that the framework incorporates token-level quantification are unsupported by original data.
The released experiment covers movies and music. The text describes 1,000 directors, 500 popular under IMDb thresholds and 500 manually curated, and 1,000 artists from an MTV list. Each anchor receives a top-25 list. Sensitive prompts insert a label before “fan of [name]” and are compared with a neutral prompt. The paper claims eight attributes and 31 values. Table 1 enumerates 30, while the released JSON contains 32: age 3, nationality 7, gender 4, continent 3, occupation 5, race 4, religion 4, and physical description 2. The artifact adds boy and girl to male and female.
Category construction has major validity problems. Boy/girl mix age and gender; American is treated as a continent; African American, black, white, and yellow mix race and ethnicity and include an offensive racial label; fat/thin are stored under physics; and Buddhist, Christian, Hinduism, and Islamic mix person adjectives with religion nouns. Prompt equivalence, naturalness, and interpretation are not human-validated. A list change may therefore arise from lexical form, semantic error, or a stereotype induced by the prompt itself rather than treatment of a well-defined group.
Released configuration also drifts from the paper. The text claims greedy search with temperature, top_p, and frequency_penalty at zero. Code sets only temperature=0 on the floating gemini-1.5-flash alias; it does not fix top_p, frequency penalty, seed, immutable revision, or execution date. The music runner defaults to 500 anchors and the notebook explicitly computes music with n=500 despite describing 1,000. The artist source has 2,999 rows rather than a documented 1,000-item selection. The director file has 1,000 rows but only 949 unique names.
Collection scripts appear to be partial process snapshots. The music runner is left with a French system instruction and a French prompt asking for movie titles even though it processes artists. The movie runner retains an English system and English template for the french folder; only the inserted label is French. Launchers acquire only French music and neutral, nationality, and French movie conditions, not every released folder. The notebook hard-codes n=500 in its main SNSR/SNSV function for every domain. The published state therefore cannot reproduce acquisition or all summaries without undocumented manual repair.
Calls did not yield complete sets. Music has 491 neutral rows and 485–494 rows per sensitive file, with different missing anchors. Movies have 975 neutral rows, only 926 unique names, and roughly 965–984 rows per condition with typically 46–49 duplicates. The parser requires number-dot-space lines. Across files it returns zero items for 155 music and 728 movie lists, with some lists parsed to 23, 24, 26, or 27 items. Analysis converts files into dictionaries by anchor, overwrites duplicates, silently catches failures, and compares only anchors surviving in both neutral and sensitive files. Effective paired n varies by group and is never reported, creating unquantified selection.
Jaccard measures overlap between neutral and sensitive top-25 lists. SNSR is the range of value-level average similarities within an attribute and SNSV their population standard deviation. These are list-dispersion measures, not direct evidence of a disadvantaged group, relevance, satisfaction, representation, exposure, allocation, or harm. The framework assumes that similarity to the neutral prompt is fairer without validating neutral output as a normative baseline or collecting preference ground truth. Legitimate personalization, noise, parser errors, and stereotyping can all create the same signal.
Formulas and code diverge. The SERP* implementation weights positions from only one list and normalizes differently from Equation 8; an identical 25-item list cannot attain the unit maximum implied by the equation. PRAG* also differs between Equation 9 and released loops. The appendix metric table calls SERP* and PRAG* higher=unfairer, while surrounding text defines them as similarity or ranking-agreement measures and later treats a drop as degradation. Direction is not internally coherent.
Tables 3 and VIII materially misassign columns. For movies, the published Jaccard sequence exactly follows released CSV columns ordered race, religion, continent, occupation, country, physical, gender, and age, but the printed header relabels them religion, continent, occupation, country, race, age, gender, and physical. Music values mainly follow religion, race, continent, country, age, gender, occupation, and physical, again under shifted labels. Thus .1363 highlighted in the abstract maps to music race in the CSV, not the printed continent label. The highlighted .0507 matches neither released music-race SNSV (.056084) nor music-continent SNSV (.042542) and is close to movie-continent (.050248), indicating further mixing.
SERP* and PRAG* discrepancies are larger. Movie Table 3 reports SERP SNSR from .0190 to .0009, whereas movie_result.csv contains .134154 to .012743 with no coherent remapping. Appendix SERP minima sometimes exceed maxima and printed SNSR is not max minus min. Published movie PRAG SNSR (.0705 to .0069) does not reproduce released values (.364045 to .026611). Some music cells follow the CSV, while others do not: published physical PRAG .0966 contrasts with released .015631. The released summaries under their implementation can be described; the full published table cannot be presumed correct.
In released summaries, Jaccard range is greatest in music for religion (.347867) and race (.136282), and in movies for race (.259877) and religion (.151257). This supports heterogeneity in overlap among author-created labels. It does not show which list is better, whether anyone is harmed, or whether the pattern is stable. There is one output per condition, a mutable endpoint alias, variable effective samples, and no interval, test, bootstrap, or multiplicity correction. The threats section says standardized statistical tests and representative datasets mitigate variation, but no statistical test is reported or implemented.
The personality claim is not implemented. Data vary demographics, occupation, nationality, body descriptors, typos, and French wording; there is no Big Five, introversion/extraversion, validated inventory, or stable profile. The appendix mentions extroverted and introverted users, but those files are absent. Big Five appears only as future work. Adding “Based on my personality” does not turn age or race into personality. PAFS has no code, intermediate data, or derived summary, leaving every table value unreproducible. Personalization utility and a formal fairness trade-off are never evaluated.
The French test is not a controlled translation: music changes both language and requested wording to movies, while movie prompts remain mostly English. Typo conditions have no repeated generations. Curves demonstrate sensitivity in those runs, not general multilingual robustness or statistically persistent unfairness.
The repository provides 84 output CSV files and enables partial audit, but has no README, license, dependency lock, tests, CI, release, or complete executable protocol. Its public ZIP also embeds a Google credential in both launchers. The value was neither used nor retained in this audit; its owner should revoke and rotate it and remove it from the artifact and history. This is a critical security and reproducibility defect.
The faithful conclusion is that a large but incomplete set of demographic prompts produced different lists from one Gemini 1.5 Flash endpoint and released CSVs show overlap heterogeneity. This is useful evidence of textual-persona sensitivity and of the need for rigorous protocols. It does not establish original predictive entropy, personality, PAFS, accuracy, disadvantaged groups, harm, significance, multilingual robustness, RecLLM generalization, or exact reproduction of the published tables.