Synthetic Personality Review

Mimetic Alignment with ASPECT: Evaluation of AI-inferred Personal Profiles

Personas, identity, and agents2026arXivApproved editorial review

Authors: Ruoxi Shang, Dan Marshall, Edward Cutrell, Denae Ford

Keywords: Persona conditioning, Psychometrics

Source: Open primary source (opens in a new tab)

4
Authors
8
Findings
17
Limitations
7
Evidence

Editorial summary

English

ASPECT uses OpenAI o1 to search 90 days of workplace meetings and messages for evidence, score 92 Communication Styles Inventory items, and present a reviewable profile with linked quotations. Among 20 employees from one organization, the initial profile agrees only modestly with self-report: 23.8% exact match, MAE 1.39, weighted kappa .34, and mean within-person correlation .39. Across 200 triads of responses to hypothetical scenarios, Profiled ranks first 42.5% of the time, compared with 32.5% for Generic and 25% for Self-Report; mean ratings are 3.33, 3.09 and 2.95 out of five. Only Profiled versus Self-Report is significant in rank tests; versus Generic, a mixed model estimates a small .24-point lift at p=.045 with substantial individual heterogeneity. The design cannot attribute this lift to the psychometric profile because Profiled also receives behavioral-evidence snippets, and the participant-corrected profile is never evaluated. Unobserved behavior is converted by default into a low score, rank analyses treat 200 repeated scenarios as independent blocks, and two qualitative counts contradict other reported values. The paper demonstrates a promising interface for auditing and negotiating inferences, not a validated personal clone, an objective personality measure, or a system ready for consequential decisions.

Español

ASPECT usa OpenAI o1 para buscar evidencia en 90 días de reuniones y mensajes de trabajo, puntuar 92 ítems del Communication Styles Inventory y mostrar al usuario un perfil revisable con citas. En 20 empleados de una sola organización, el perfil inicial coincide de forma modesta con el autoinforme: 23,8% exacto, MAE 1,39, kappa ponderado 0,34 y correlación media intrapersona 0,39. En 200 tríadas de respuestas a escenarios hipotéticos, la condición Profiled queda primera 42,5% de las veces, frente a 32,5% Generic y 25% Self-Report; las valoraciones medias son 3,33, 3,09 y 2,95 sobre cinco. Solo Profiled frente a Self-Report es significativo en rangos; frente a Generic, el modelo mixto estima una mejora pequeña de 0,24 puntos con p=0,045 y gran heterogeneidad individual. El diseño no permite atribuirla al perfil psicométrico porque Profiled recibe además fragmentos de evidencia conductual, y nunca evalúa el perfil corregido por el participante. La ausencia de conducta observada se convierte por defecto en una puntuación baja, los análisis de rangos tratan 200 escenarios repetidos como bloques independientes y hay contradicciones internas en dos conteos cualitativos. El trabajo demuestra una interfaz prometedora para auditar y negociar inferencias, no un clon personal validado, una medida objetiva de personalidad ni un sistema listo para decisiones reales.

Research question

Whether an LLM can infer a detailed communicative style profile from workplace communication, how it differs from self-report and is negotiated by the user, and whether responses conditioned by that profile seem more like the user than generic or self-report-based alternatives.

Method

A locally installed Flask application processes 90 days of meeting transcriptions, group messages, and direct messages. OpenAI o1 searches for two to five pieces of evidence per facet, scores 92 items on a 1-5 scale with justification, and assigns an absence score when it finds no evidence. Each participant self-rates, compares both profiles, and can revise the result. Then they rank and rate, blind and in random order, three responses to ten scenarios: Generic only sees the scenario, Self-Report receives a description derived from the self-report, and Profiled receives the initial ASPECT profile plus compact evidence fragments.

Sample: 20 employees from a large organization: 11 men and 9 women, 18-54 years, from internships to senior positions; 9 worked in office, 10 in hybrid mode, and 1 remotely. They describe themselves as technically sophisticated and comfortable with AI. Each contributed 90 days of workplace communication, completed a two-hour session, and received 100 USD. The evaluation gathers 1,840 item pairs, 411 facet judgments, and 200 participant-scenario combinations with three responses each.

Findings

  • Across 1,840 pairs, exact AI/self-report agreement is 23.8%, MAE 1.39 [1.34-1.45], weighted kappa 0.34, and mean intraperson Spearman 0.39 [0.31-0.44].
  • Intraperson medians rise from 0.46 per item to 0.55 per facet and 0.72 per dimension, while between-person associations tend to approach zero with n=20.
  • Evidence is missing for Angriness and Defensiveness in 19/20 people, Nonsupportiveness in 18, Concealingness and Derogatoriness in 17, and Authoritarianism in 15.
  • The 411 evaluations are distributed as 141 aligned, 142 disapprove, 42 middle ground, 31 approve AI, and 55 unsure; 73/411, or 17.7%, correspond to approve AI plus middle ground.
  • Profiled ranks first 85/200 times (42.5%), Generic 65 (32.5%), and Self-Report 50 (25%); mean ranks are 1.84, 2.00, and 2.15.
  • The global Friedman test gives chi-square 9.31, p=0.0095, and W=0.023; after Holm, only Profiled-Self-Report is significant (p=0.0067, r=0.22).
  • Mean ratings are 3.33 for Profiled, 3.09 Generic, and 2.95 Self-Report; the mixed model estimates Profiled-Generic beta 0.24, p=0.045, and Self-Report-Generic beta -0.14, p=0.26.
  • The random slope deviations, 0.87 for Self-Report and 0.94 for Profiled, show that individual variation is large relative to the small mean effect.

Limitations

  • Self-report and AI inference measure different perspectives and neither is ground truth; accepting or negotiating an inference does not prove it is correct.
  • Profiled receives ratings plus evidence fragments and Self-Report only ratings; the effect of the profile is not separated from behavioral detail, content, or lexical copying.
  • The scenarios use the initial ASPECT profile, not the revised one; the behavioral benefit of inspection, correction, and human control remains unevaluated.
  • Assigning an absence score when there is no evidence confounds unobserved behavior with a low trait, especially in professional communication.
  • Wald intervals for first positions and Friedman/Wilcoxon use 200 participant-scenario pairs as blocks, ignoring that they cross within only 20 people and 10 scenarios.
  • The mixed model simplifies its structure after convergence problems; the Profiled-Generic contrast is small and borderline, with no bootstrap or specification sensitivity.
  • The MAE interval method and kappa weighting type are not explained; ICC uncertainty, item-facet-person hierarchy, and multiplicity are missing.
  • Section 5.1.5 says middle ground is 99/411=24%, but the published taxonomy contains 42/411=10.2%.
  • The prose classifies participant preferences as 9 Profiled, 2 Self-Report, and 9 mixed/weak; Table 4 reports 10 Profiled, 4 Generic, 2 Self-Report, and 4 Mixed.
  • Three coders review an unspecified subset, one person verifies the total, there is no intercoder reliability, and the categories are approximate and post hoc.
  • The ten scenarios are hypothetical, a single instance per configuration; they do not evaluate real behavior, consequences, temporal stability, or prolonged deployment.
  • Exact snapshot of o1, seeds, temperature, top-p, runs, context limits, truncation, full prompts, and stability across executions are missing.
  • There is no public code, data, materials, outputs, environment, or analysis scripts; the official pages only link the preprint.
  • The small sample, from one organization and comfortable with AI, does not allow generalization or auditing of race, ethnicity, culture, language, disability, seniority, or fairness.
  • Messages and meetings contain third-party data; consent/redaction of colleagues, processing route by o1, provider retention, deidentification, or security controls are not detailed.
  • Hallucinations and erroneous attributions are acknowledged, but frequency, severity, citation accuracy, or harm are not quantified.
  • A workplace communication profile can be repurposed for surveillance, evaluation, hiring, or discipline; no technical or governance barriers against such uses are defined.

What the study does not establish

  • It does not demonstrate that ASPECT objectively measures personality or stable style, or that o1 is more correct than the participant.
  • It does not attribute preference to the psychometric profile, because the Profiled condition additionally has behavioral evidence.
  • It does not prove that revising and correcting the profile improves responses: the negotiated profile never enters the subsequent evaluation.
  • It does not establish a robust advantage over Generic in rankings; in ratings the difference is small and p=0.045.
  • It does not demonstrate that the system benefits all people; random slopes show strong heterogeneity.
  • It does not convert absence of evidence into valid evidence of absence of a trait.
  • It does not validate a personal clone, value alignment, real decisions, safety, or longitudinal behavior.
  • It does not justify organizational deployment or high-impact decisions with n=20, one organization, and hypothetical scenarios.
  • It does not resolve third-party privacy, provider processing, deidentification, retention, or possible workplace surveillance.
  • It does not allow reproducing profiles and results from available public artifacts.

Traceability

Scope: Full text

Version: arXiv:2603.26922v1

Consulted source: https://arxiv.org/abs/2603.26922v1

Review: Codex 20-page visual full-text, official arXiv metadata, Microsoft Research, author-page, GitHub artifact, construct, hierarchical-statistical, privacy, ethics and reproducibility audit, 2026-07-17

Approval: Codex fidelity pass, 2026-07-17

English translation: approved, 2026-07-18

Models evaluated

  • OpenAI o1

Instruments and metrics

  • Communication Styles Inventory: 92 items, 6 dimensions and 23 facets
  • Comparación de rating IA/autoinforme con evidencia enlazada
  • Codificación cualitativa de 411 evaluaciones de faceta
  • 10 escenarios laborales hipotéticos personalizados
  • Ranking ciego aleatorizado de tres condiciones
  • Escala sounds-like-me de cinco puntos
  • Friedman, Wilcoxon-Holm y Kendall W
  • Modelo lineal mixto con interceptos de participante y escenario

Data used

  • Participant-exported prior 90 days of workplace meeting transcripts, group messages and direct messages
  • Participant self-reported Communication Styles Inventory ratings
  • ASPECT evidence, item ratings, review decisions and hypothetical-scenario judgments

Evidence and location

  • ASPECT pipeline, CSI, evidence, absence scoring, and revision: arXiv v1, pp. 1-7, Figures 1-3; all 20 PDF pages visually inspected
  • Sample, workplace data, procedure, ethics, and compensation: arXiv v1, pp. 4-6
  • AI/self-report agreement, evidence density, and qualitative analysis: arXiv v1, pp. 7-11, Figures 4-5 and Tables 1-2
  • Ranking, ratings, mixed model, heterogeneity, and count contradictions: arXiv v1, pp. 11-15, Tables 3-5
  • Scenario, style, and evidence prompts: arXiv v1, pp. 18-20, Appendix
  • Metadata, preprint status, and absence of public artifact: Official arXiv Atom/HTML, Microsoft Research page, first-author page and GitHub repository APIs inspected 2026-07-17
  • Comprehensive audit of construct, ground truth, confound, statistics, privacy, and reproducibility: reports/verification/article-382-aspect-profile-ground-truth-evidence-confound-statistics-privacy-and-reproducibility-audit.json