Synthetic Personality Review

PerFairX: Is There a Balance Between Fairness and Personality in Large Language Model Recommendations?

Applications, bias, and safety2025IEEEApproved editorial review

Original title: PerFairX: Is There a Balance Between Fairness and Personality in LLM Recommendations?

Authors: Chandan Kumar Sah, Xiaoli Lian

Keywords: personality-based recommendation, fairness evaluation, OCEAN model, LLM recommender systems, demographic equity, personalization trade-offs, zero-shot learning

Source: Open primary source (opens in a new tab)

2
Authors
48
Findings
118
Limitations
22
Evidence

Editorial summary

English

Sah and Lian present PerFairX, an exploratory framework for comparing OCEAN-trait personalization with demographic disparity in LLM-generated recommendations. The study infers personality profiles from genre preferences, selects five MovieLens 10M users and five Last.fm 360K users, and asks ChatGPT and DeepSeek for top-15 lists under a generic prompt and a personality-sensitive prompt. It defines ten metrics, although only eight enter the final comparison and they are aggregated with equal weights. DeepSeek's Personality Alignment Score rises from 0.280 to 0.848 on MovieLens and from 0.315 to 0.872 on Last.fm; ChatGPT instead declines slightly from 0.749 to 0.739 and from 0.782 to 0.728. For both models and domains, genre-personality alignment falls and DP and EO increase under the sensitive prompt, while intra-list diversity improves. The reported FPx scores favor DeepSeek, but they depend on an unvalidated equal-weight sum with no uncertainty analysis. The evidence does not support a reproducible benchmark: exact model snapshots, repeated runs, code, derived data, and the title-matching protocol are absent. MovieLens 10M also states officially that it contains no demographic information, yet the paper computes gender, age, and occupation without explaining another source; and the prose calls ChatGPT less biased even though its own tables report lower DP and EO for DeepSeek. PerFairX offers useful vocabulary for discussing personalization-equity tensions, but this small, circular, and internally contradictory experiment does not validate psychological personality, causal fairness, or general model superiority.

Español

Sah y Lian presentan PerFairX, un marco exploratorio para comparar personalización por rasgos OCEAN y disparidad demográfica en recomendaciones generadas por LLM. El estudio infiere perfiles de personalidad a partir de preferencias de género, selecciona cinco usuarios de MovieLens 10M y cinco de Last.fm 360K, y solicita a ChatGPT y DeepSeek listas top-15 con un prompt genérico y otro sensible a la personalidad. Define diez métricas, aunque solo informa ocho en la comparación final y las agrega con pesos iguales. DeepSeek aumenta su Personality Alignment Score de 0,280 a 0,848 en MovieLens y de 0,315 a 0,872 en Last.fm; ChatGPT, en cambio, baja ligeramente de 0,749 a 0,739 y de 0,782 a 0,728. En ambos modelos y dominios disminuye la alineación género-personalidad y aumentan DP y EO con el prompt sensible, mientras mejora la diversidad intralista. Los FPx publicados favorecen a DeepSeek, pero dependen de una suma no validada con pesos iguales y sin incertidumbre. La evidencia no sostiene un benchmark reproducible: no se identifican snapshots de los modelos, ejecuciones repetidas, código, datos derivados ni protocolo de emparejamiento de títulos. Además, MovieLens 10M declara oficialmente que no contiene datos demográficos, aunque el artículo calcula género, edad y ocupación sin explicar otra fuente; y el texto afirma que ChatGPT tiene menor sesgo pese a que sus propias tablas muestran DP y EO menores para DeepSeek. PerFairX aporta un vocabulario útil para discutir tensiones entre personalización y equidad, pero este experimento pequeño, circular y contradictorio no valida personalidad psicológica, equidad causal ni superioridad general de un modelo.

Research question

How does the alignment between recommendations and inferred OCEAN traits change when ChatGPT and DeepSeek receive neutral or personality-sensitive prompts, what effect does that change have on demographic disparities and intra-list diversity, and which model obtains the best trade-off according to the PerFairX aggregate?

Method

The article filters users with at least 200 interactions, infers a five-dimensional OCEAN vector through genre affinity, score dispersion, temporal activity and catalog diversity, and manually links genres such as science fiction, documentary, romance, indie or jazz with traits. It selects five profiles from MovieLens 10M and five from Last.fm 360K. For each profile it queries ChatGPT and DeepSeek with a neutral prompt and a personalized one and obtains fifteen recommendations. It calculates PAS and GPA for personality fit; DP, EO and ILF for disparity or diversity; SNSR, SNSV and Jaccard for prompt sensitivity; and Precision@15 and Recall@15 for relevance. The published comparison shows eight metrics and constructs FPx by summing PAS, GPA, 1-DP, 1-EO, ILF, Jaccard, Precision and Recall with equal weights. Separately, Figure 1 reports a social media survey with 120 people on a single vignette of three celebrities and compares their choice with four not fully specified LLMs.

Sample: The main evaluation uses ten profiles: five from MovieLens 10M and five from Last.fm 360K. With two models and two prompt conditions, the design implies forty top-15 lists if each combination was run once; the paper does not report repetitions, seeds or final number of calls. Table 1 describes 17.4 thousand women and 43.6 thousand men in MovieLens and 2 thousand women and 8 thousand men in Last.fm, but does not explain how it obtains those subsets. The official source of MovieLens 10M registers 71,567 users and no demographic attribute; the official source of Last.fm 360K registers 359,347 users and 17,559,530 lines, well above the 598 thousand records declared in Table 1. The introductory study of Figure 1 recruits 120 people through LinkedIn, Facebook, X and WeChat; it does not report their characteristics, consent, compensation or duplicate control.

Findings

  • The work was published in IEEE/CVF ICCV Workshops 2025, Multimodal Continual Learning, pages 2771-2780, with DOI 10.1109/ICCVW69036.2025.00289.
  • The open CVF copy declares itself identical to the accepted version except for the watermark.
  • The final inspected PDF has ten pages and sha256 2c7a7a66a12e37db97a904922746dabe0fa98ace5ee256ea58a7d3e31ac55515.
  • The final version and arXiv v1 preserve the same study; the tokenized comparison reaches 0.9433 and the observed differences are mainly typographical, layout and metadata.
  • The official CVF record and BibTeX list Chandan Kumar Sah and Xiaoli Lian, but the visible title page, arXiv, DBLP and the Crossref record of the DOI only show Sah; the internal PDF metadata does include both names.
  • The main experiment does not evaluate the entire population of the datasets: it uses five movie profiles and five music profiles.
  • Each profile receives a neutral prompt and a personality-sensitive one to produce fifteen recommendations with ChatGPT and DeepSeek.
  • The ChatGPT link points to GPT-4o documentation, but does not identify a dated model, API version or snapshot.
  • DeepSeek is identified only through the generic documentation of its API.
  • The OCEAN vectors do not come from a questionnaire: they are inferred from genre affinity, rating dispersion, temporal activity and catalog diversity.
  • The article manually associates science fiction and documentary with Openness, romance with Agreeableness, indie and jazz with Openness and documentary or classical with Conscientiousness.
  • PAS projects the recommended genres back onto a trait vector constructed with that same type of association, creating a circular evaluation.
  • The framework defines ten metrics: PAS, GPA, DP, EO, ILF, SNSR, SNSV, Jaccard, Precision and Recall.
  • The results do not present SNSR or SNSV values.
  • In the results section SNSR and SNSV change name to Semantic Novelty Similarity Ratio and Semantic Novelty Sensitivity Variation, without reconciling this with their definitions.
  • Table 4 compares eight metrics and Figure 5 only plots seven, although the text speaks of eight.
  • In MovieLens, DeepSeek's PAS rises from 0.280 with neutral prompt to 0.848 with sensitive prompt.
  • In Last.fm, DeepSeek's PAS rises from 0.315 to 0.872.
  • ChatGPT's PAS does not improve: it drops from 0.749 to 0.739 in MovieLens and from 0.782 to 0.728 in Last.fm.
  • Therefore, the general claim that sensitive prompting significantly improves alignment does not hold for ChatGPT in Table 2 itself.
  • GPA decreases under the sensitive prompt in all four combinations: 0.709 to 0.407; 0.713 to 0.336; 0.695 to 0.421; and 0.731 to 0.352.
  • DP and EO increase with the sensitive prompt for both models and datasets, which the protocol interprets as greater demographic disparity.
  • ILF increases in all four combinations; the largest change is DeepSeek-MovieLens, from 0.475 to 0.968.
  • Table 4 shows lower DP and EO for DeepSeek than for ChatGPT under the sensitive prompt in both domains.
  • The subsequent prose claims that ChatGPT maintains lower disparity in DP and EO, in direct contradiction with Table 4.
  • In sensitive MovieLens, DP/EO are 0.825/0.952 for ChatGPT and 0.726/0.901 for DeepSeek.
  • In sensitive Last.fm, DP/EO are 0.801/0.935 for ChatGPT and 0.711/0.884 for DeepSeek.
  • DeepSeek obtains higher Precision@15 and Recall@15, but the absolute values remain low: at most 0.165 and 0.045.
  • The FPx aggregate uses PAS + GPA + (1-DP) + (1-EO) + ILF + Jaccard + Precision + Recall with equal weights.
  • The four FPx of Table 5 are reproduced exactly with that sum: 1.994 and 2.895 in MovieLens; 2.022 and 2.961 in Last.fm.
  • DeepSeek's superiority in FPx depends largely on the large ILF difference and a choice of weights without validation or sensitivity analysis.
  • Table 4 marks Jaccard with a downward arrow, but the definition and the FPx formula treat a higher Jaccard as more stability and higher score.
  • No deviations, intervals, p-values or tests supporting the use of 'significantly' are published.
  • The text claims that perceptions of fairness, trust and satisfaction change, but the main experiment does not measure any of those three constructs with people.
  • MovieLens 10M officially declares that it contains no demographic information, while Table 1 attributes gender, age and occupation without describing an external source or link.
  • The 61,000 users per sex that Table 1 attributes to MovieLens cannot all be the set filtered to 200 interactions: the minimum would be 12.2 million ratings, above the 10,000,054 of the complete dataset.
  • The official source of Last.fm 360K reports 359,347 users and 17,559,530 lines; Table 1 uses 10,000 users per sex and 598,000 records without explaining the reduction.
  • It is not explained how titles or artists in free text from the LLMs are linked to dataset identifiers to calculate Precision and Recall.
  • The neutral prompt asks for popular content for a general audience, while the sensitive one asks for individual fit; the comparison simultaneously changes popularity, target and profile.
  • The sensitive examples also include age and gender, so they do not isolate the effect of personality.
  • Figure 3 labels romance lists for a young and agreeable woman as personalized but unfair, without demonstrating that gender narrowness equates to demographic discrimination.
  • Figure 1 reports that 80% of 120 evaluators chose Elon as the least likely consumer of Interstellar.
  • The text claims that four LLMs ignored that choice, but Figure 1 itself shows that DeepSeek selected Elon.
  • The human majority of a stereotyped vignette does not constitute a real observed preference or a psychological truth.
  • There is no link to code, complete prompts, raw outputs, profiles, partitions or reproducible results.
  • The article does not describe multimodal experiments or continual learning despite being published in that workshop and using those terms in the conclusion.
  • The acknowledgments declare support from the Software Engineering Institute of Beihang University and a Chinese Government Scholarship.
  • The work raises relevant questions, but the published evidence does not reach the standard of a validated and reproducible benchmark.

Limitations

  • The main evaluation uses only ten profiles.
  • It is not justified why five profiles per domain are representative.
  • The identity or complete vector of the ten profiles is not published.
  • The profile sampling procedure is not reported.
  • A selection seed is not reported.
  • Sampling is not repeated with other profiles.
  • It is not reported how many executions were performed per combination.
  • Temperature, top-p, seed or decoding parameters are not reported.
  • The date of the API calls is not reported.
  • The exact snapshot of GPT-4o or ChatGPT is not identified.
  • The exact DeepSeek model is not identified.
  • It is not reported whether failed or unparseable responses were regenerated.
  • The system prompt is not published.
  • Only partial examples of sensitive prompts are shown.
  • Figure 3 presents more than one neutral and sensitive wording, without clarifying which produced the tables.
  • The neutral prompt and the sensitive one do not differ only in personality.
  • The neutral prompt optimizes popularity and generality.
  • The sensitive prompt also changes tone, preferences, exclusions and in some examples demographic attributes.
  • The comparison does not allow causally attributing the differences to OCEAN.
  • Personality is inferred from behavior without an observed psychometric instrument.
  • The complete equation that transforms interactions into each OCEAN dimension is not published.
  • Weights for genre affinity, dispersion, temporal activity and diversity are not published.
  • Thresholds for determining dominant traits are not published.
  • The complete table of correspondences between genres and traits is not published.
  • The genre-trait correspondences rely on population associations and are applied as if they described individuals.
  • The inferred vector is not validated against users' OCEAN self-report.
  • Test-retest reliability of the profiles is not reported.
  • PAS evaluates with the same family of associations used to construct the profile.
  • GPA shares the same circularity between inferred traits and recommended genres.
  • A high PAS score may reflect obedience to prompt words and not stable personality.
  • The study does not measure model behavior outside the recommendation task.
  • Stability of the supposed personality across sessions or reformulations is not tested.
  • MovieLens 10M does not contain the demographic attributes used by the analysis.
  • An auxiliary dataset or demographic enrichment process for MovieLens is not identified.
  • The counts in Table 1 are not reconciled with the official MovieLens documentation.
  • The 200-interaction threshold is not reconciled with the 61,000 users per sex declared.
  • It is not reported how many users survive the 200-interaction filter.
  • The proportion of missing demographic data is not reported.
  • It is not explained how the senior and young categories are formed.
  • It is not explained what ages fall outside those two categories.
  • It is not explained how countries or continents are grouped.
  • It is not explained how occupation is used in the binary DP and EO comparisons.
  • The Last.fm counts and the total records do not match the official source without the subset being documented.
  • The processed Last.fm partition is not published.
  • It is not reported how profiles with empty gender, age or country are handled.
  • The event Y-hat=1 for a free-text-generated recommendation list is not precisely defined.
  • The true relevance Y used in EO is not precisely defined.
  • A train-test or temporal partition for Precision and Recall is not described.
  • It is not reported how many relevant items each profile has.
  • Normalization of titles, aliases, remakes or artists is not described.
  • It is not reported how many recommendations do not belong to the catalog.
  • It is not reported how title hallucinations are treated.
  • It is not reported how duplicates in top-15 are resolved.
  • It is not specified whether ILF uses items, genres or a normalized distribution of categories.
  • Intra-list diversity entropy is not by itself a measure of equity.
  • The term Intra-list Fairness may confuse content diversity with fair treatment between people.
  • Exposure of providers or creators is not measured.
  • Harm, utility or real preference for protected groups is not studied.
  • Comparability of preferences before applying DP is not checked.
  • DP may penalize legitimate taste differences and the article does not resolve that problem.
  • EO requires relevance ground truth, but the protocol to obtain it is not published.
  • The framework does not present a measure of PAS disparity between personality groups.
  • Therefore, the so-called psychological fairness is reduced mainly to alignment, not to equity between psychological types.
  • SNSR and SNSV are defined but not reported.
  • The names of SNSR and SNSV change between method and results.
  • The Jaccard arrow in Table 4 contradicts its interpretation of stability and its sign in FPx.
  • Figure 5 omits Jaccard despite the announced comparison of eight metrics.
  • Per-user results are not published.
  • Per-group distributions are not published.
  • Standard deviations are not published.
  • Confidence intervals are not published.
  • P-values are not published.
  • No statistical test is performed for the significance claims.
  • No correction for multiple comparisons is applied.
  • Sensitivity to alternative prompts is not analyzed.
  • Sensitivity to the weights of the FPx aggregate is not analyzed.
  • The eight components of FPx receive equal weights without empirical justification.
  • The aggregate combines constructs with different interpretations and scales.
  • DeepSeek's large ILF dominates part of the composite difference.
  • A single scalar hides the real conflict between DP, EO, alignment, diversity and accuracy.
  • FPx is not compared with a human evaluation of utility or fairness.
  • It is not compared with a traditional collaborative recommender.
  • It is not compared with a reproducible popularity recommender.
  • It is not compared with a non-psychological personalization technique under the same budget.
  • Gemini, Claude or Llama are not evaluated despite being cited as future work.
  • The conclusions about DeepSeek and ChatGPT are based on unidentified versions and may not persist.
  • The phrase that ChatGPT has lower DP and EO contradicts Table 4.
  • The conclusion of outputs perceived-as-fair does not come from a human evaluation of perception.
  • The study does not measure trust or satisfaction despite claiming so in the introduction.
  • Figure 1 uses a single movie and three famous people constructed through stereotypes.
  • The traits of Figure 1 do not come from self-reports of the represented people.
  • The real cinematic preference of those people is not observed.
  • The human vote is treated as ground truth without validation.
  • Age, gender, country or language of the 120 evaluators is not reported.
  • Compensation is not reported.
  • Consent is not reported.
  • Ethical review or exemption is not reported.
  • Control of duplicate responses across four social networks is not reported.
  • The exact text of the questionnaire or the order of options is not reported.
  • Figure 1 shows that DeepSeek agrees with the human majority, contrary to the claim that the four LLMs fail.
  • The complete responses of the four LLMs in the case are not published.
  • No code is offered.
  • No configuration files are offered.
  • No processed profiles are offered.
  • No raw recommendations are offered.
  • No intermediate metric calculations are offered.
  • No artifact reproducing the tables is offered.
  • No license for code or derived data is declared.
  • There is no preregistration.
  • No independent replication is performed.
  • Languages other than English are not studied.
  • Domains other than movies and music are not studied.
  • Longitudinal user behavior is not studied.
  • Change in personality or preference over time is not studied.
  • Continual learning is not implemented.
  • A multimodal system is not implemented.
  • Placement in a continual multimodal learning workshop does not constitute evidence on those two topics.
  • The authorship discrepancy between CVF, visible PDF, Crossref, arXiv and DBLP remains without editorial explanation.

What the study does not establish

  • It does not demonstrate that LLMs possess psychological OCEAN traits.
  • It does not demonstrate that the inferred vector describes the real personality of users.
  • It does not validate the correspondences between genres and traits at the individual level.
  • It does not demonstrate stable personality across prompts, sessions or tasks.
  • It does not demonstrate that improving PAS improves user satisfaction or well-being.
  • It does not demonstrate that ILF is equity.
  • It does not demonstrate causal discrimination produced by personality.
  • It does not demonstrate that DP and EO differences are due only to the personality prompt.
  • It does not demonstrate demographic equity on MovieLens 10M with the published source.
  • It does not demonstrate psychological fairness between personality types.
  • It does not demonstrate that PerFairX is a reproducible benchmark.
  • It does not demonstrate that the FPx aggregate represents valid social preferences over trade-offs.
  • It does not demonstrate that equal weights are the correct aggregation.
  • It does not demonstrate a statistically significant improvement.
  • It does not demonstrate that DeepSeek is generally a better recommender.
  • It does not demonstrate that ChatGPT is fairer by DP or EO.
  • It does not demonstrate that ChatGPT is perceived as fairer by users.
  • It does not demonstrate user trust or satisfaction.
  • It does not demonstrate that the majority of 120 votes is a true preference of the represented celebrities.
  • It does not demonstrate that four LLMs fail the Figure 1 case.
  • It does not demonstrate generalization to other models, versions, cultures or domains.
  • It does not demonstrate operation in continual learning or multimodal systems.
  • It does not justify deploying the framework as a fairness audit without resolving demographic provenance, validity and reproducibility.

Traceability

Scope: Full text

Version: ICCVW 2025 Multimodal Continual Learning workshop open-access accepted version, pp. 2771-2780; DOI 10.1109/ICCVW69036.2025.00289; CVF copy identical to accepted version apart from watermark

Consulted source: https://openaccess.thecvf.com/content/ICCV2025W/MCL/html/Sah_PerFairX_Is_There_a_Balance_Between_Fairness_and_Personality_in_ICCVW_2025_paper.html

Review: Codex full-text, peer-reviewed-version, bilingual-fidelity, visual, bibliographic, dataset-provenance, psychometric-validity, fairness-operationalization, metric-recalculation, internal-consistency, reproducibility and ethics audit, 2026-07-15

Approval: Codex fidelity pass, 2026-07-15

English translation: approved, 2026-07-18

Models evaluated

  • ChatGPT, linked to GPT-4o documentation but without an exact model snapshot
  • DeepSeek API, exact model and snapshot not reported
  • Four incompletely specified LLMs in the Figure 1 case study, visually represented as xAI, ChatGPT, DeepSeek and Gemini

Instruments and metrics

  • Five-dimensional OCEAN profile inferred from behavioral proxies
  • Personality Alignment Score (PAS)
  • Genre-Personality Alignment (GPA)
  • Demographic Parity difference (DP)
  • Equal Opportunity difference (EO)
  • Intra-list Fairness or genre-diversity entropy (ILF)
  • Sensitive-to-Neutral Similarity Range (SNSR@K)
  • Sensitive-to-Neutral Similarity Variance (SNSV@K)
  • Jaccard@15 between neutral and sensitive lists
  • Precision@15 and Recall@15
  • Equal-weight PerFairX aggregate (FPx)
  • Single-question social-media case-study vote

Data used

  • MovieLens 10M
  • Last.fm Dataset 360K
  • Ten selected or constructed user profiles
  • Top-15 recommendation lists generated under two prompt conditions
  • Figure 1 social-media responses from 120 evaluators

Evidence and location

  • Final bibliographic identity: IEEE/CVF ICCVW 2025, Multimodal Continual Learning workshop, pp. 2771-2780; DOI 10.1109/ICCVW69036.2025.00289; IEEE document 11375720
  • Open full text inspected: .cache/editorial-sources/article-085/source.pdf; CVF accepted version; 10 pages; sha256 2c7a7a66a12e37db97a904922746dabe0fa98ace5ee256ea58a7d3e31ac55515
  • Relationship between final version and preprint: CVF final PDF versus preserved .cache/editorial-sources/article-085/versions/arxiv-v1.pdf; normalized token sequence ratio 0.9433384379785605
  • Authorship discrepancy: CVF landing page and PDF metadata list Chandan Kumar Sah and Xiaoli Lian; visible title page, arXiv v1, DBLP and Crossref DOI metadata list Chandan Kumar Sah
  • Research question and proposed framework: Abstract and Sections 1-3, final pp. 2771-2775
  • Survey of 120 people and Figure 1 contradiction: Section 1 and Figure 1, final pp. 2771-2772: prose says four LLMs overlooked Elon; figure shows DeepSeek selecting Elon
  • Definition of ten metrics: Sections 3.1.3 and 3.2, equations 1-11, final pp. 2773-2775
  • OCEAN inference and prompts: Sections 3.3-3.5, final p. 2775
  • Sample of five profiles per domain: Section 4, final p. 2775
  • Declared dataset statistics and attributes: Table 1, final p. 2776
  • MovieLens 10M lacks demographics: Official GroupLens MovieLens 10M README: 10,000,054 ratings, 10,681 movies, 71,567 users and no demographic information
  • Official Last.fm 360K statistics: Official UPF Music Technology Group Last.fm 360K version 1.2 page: 17,559,530 lines, 359,347 unique users, profile fields gender, age, country and signup
  • Alignment results: Table 2 and Section 4.1, final p. 2776
  • DP, EO and ILF results: Table 3 and Section 4.2, final p. 2776
  • Qualitative confusion between personalization and fairness: Figure 3 and its interpretation, final p. 2777
  • Model comparison and DP/EO contradiction: Table 4 versus Section 4.3 prose, final pp. 2777-2778
  • FPx aggregate reproduced: Equation 11 and Table 5, final pp. 2775 and 2778; equal-weight sums reproduce 1.994, 2.895, 2.022 and 2.961
  • Omitted metrics and Jaccard inconsistency: Definitions on p. 2774, Table 4 and Figure 5 on p. 2778: SNSR/SNSV absent; Jaccard marked lower-is-better but added positively to FPx
  • Conclusions and scope claims: Section 5, final p. 2778
  • Declared funding: Section 6 Acknowledgments, final p. 2779
  • Absence of reproducible artifacts: Complete final PDF, link annotations and current web search inspected 15 Jul 2026: model documentation links only; no paper code or data repository found
  • Integral reading and visual verification: All 10 final pages rendered and inspected at original resolution, including Figures 1-5, Tables 1-5, equations 1-11, acknowledgments and references; checked 15 Jul 2026