Synthetic Personality Review

Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks

Evaluation and psychometric validity2026ICML ProceedingsApproved editorial review

Authors: Peiyu Li, Xiuxiu Tang, Si Chen, Ying Cheng, Ronald Metoyer, Ting Hua, Nitesh V. Chawla

Keywords: Large Language Models, Adaptive Testing, Psychometrics, Item Response Theory, Model Evaluation

Source: Open primary source (opens in a new tab)

7
Authors
40
Findings
99
Limitations
15
Evidence

Editorial summary

English

ATLAS is an ICML 2026 Spotlight paper proposing to replace static model evaluation with Item Response Theory-based computerized adaptive testing. This review uses the complete 24-page v3 paper, visually inspects all 24 pages, and audits the official repository at commit c0bc19b. The repository releases parameters, results, and an 87 MB ZIP containing response matrices, although the README incorrectly says those matrices are not committed.

The study starts from precomputed Open LLM Leaderboard responses for WinoGrande, TruthfulQA, HellaSwag, GSM8K, and ARC. It retains complete-response models, removes the bottom 0.1 percentile, and performs a ten-bin stratified 90/10 model calibration/test split. It filters nearly constant items, items above 95% accuracy, and items with point-biserial correlation below 0.1. It fits partitioned 3PL IRT models, attempts to align them through common-person mean–sigma linking, and obtains a whole-bank WLE reference ability over the filtered bank. For each test model, ATLAS starts at θ=0, samples among the five highest-Fisher-information items, updates θ by EAP, and stops after at least 30 items when SE falls below 0.1, 0.2, or 0.3, with a maximum of 500. It does not run the LLMs; it looks up selected-item responses in an already complete response matrix.

In the published tables, ATLAS uses an average of 30–89 items. WinoGrande, TruthfulQA, and HellaSwag ability MAEs are 0.155/0.166/0.179, 0.064/0.073/0.071, and 0.157/0.163/0.165 at the three thresholds. On GSM8K, ATLAS does not beat MetaBench-Secondary in absolute MAE: ATLAS is 0.150–0.177 versus 0.096; on ARC, the 0.1 threshold does obtain the lowest MAE, 0.084. IES systematically favors ATLAS because it multiplies relative error by item count relative to Random100; it can improve even while absolute error worsens. p-IRT reconstruction remains below five percentage points MAE, but it blends observed responses with IRT probabilities for unadministered items and compares the result with raw accuracy over a bank that, for several benchmarks, contains items excluded from calibration. Generalization experiments cover ARC only: strict Mixtral-family holdout yields MAE 0.091–0.109, while the temporal split worsens to 0.126–0.162.

The code audit finds a problem at the core of the validation. If the linked scale is θref=A·θk+B and mirt uses predictor a·θ+d, the curve-preserving transformation is a*=a/A and d*=d−aB/A. The code and README instead use d*=A·d+B·a. The linked intercepts therefore do not represent the same items on the reference scale. The script's post-link correlation does not detect this because correlation between θref and an affine transform of θk is invariant to shift and scale. Since whole-bank WLE, adaptive selection, p-IRT probabilities, and most baselines use the combined parameters, this error contaminates the reference ATLAS is said to match.

Two further reporting errors are material. Table 1 labels TinyBenchmarks values from 364.24 to 646.82 as RMSEA even though RMSEA cannot be on that scale; the released TinyBenchmarks fitting script does not calculate M2 or RMSEA at all. Average item exposure is also computed only over items selected at least once, omitting zeros despite the paper's formula averaging over the entire bank. For HellaSwag at SE≤0.1, 3.86% is reported but the correct full-bank mean over 5,600 items is 0.73%; 4,542 items are never used and the most exposed item reaches 40.67%. GSM8K maximum exposure reaches 56.43%. The published statistic therefore does not establish broad utilization or prevent concentration. ATLAS is a promising idea with unusually substantial artifacts, but its numbers do not yet validate a production-ready replacement for static benchmarks; the linking must be corrected, results rerun, and prospective cost, content, and external validity evaluated.

Español

ATLAS es un trabajo aceptado como Spotlight en ICML 2026 que propone sustituir la evaluación estática de modelos por computerised adaptive testing basado en teoría de respuesta al ítem. Esta revisión usa la versión v3 completa de 24 páginas, inspecciona visualmente sus 24 páginas y audita el repositorio oficial en el commit c0bc19b. El repositorio publica parámetros, resultados y un ZIP de 87 MB con matrices de respuesta, aunque el README afirma por error que esas matrices no están incluidas.

El estudio parte de respuestas ya calculadas del Open LLM Leaderboard para WinoGrande, TruthfulQA, HellaSwag, GSM8K y ARC. Retiene modelos con respuestas completas, elimina la cola inferior por debajo del percentil 0,1 y divide los modelos en diez estratos para un 90% de calibración y 10% de prueba. Filtra ítems casi constantes, con más de 95% de aciertos o correlación punto-biserial inferior a 0,1. Ajusta modelos IRT 3PL en particiones, intenta enlazarlas mediante common-person mean–sigma linking y obtiene una habilidad de referencia con WLE sobre el banco filtrado. Para cada modelo de prueba, ATLAS comienza en θ=0, selecciona entre los cinco ítems de mayor información de Fisher, actualiza θ por EAP y se detiene tras al menos 30 ítems cuando el error estándar baja de 0,1, 0,2 o 0,3, con máximo 500. No ejecuta los LLM: busca las respuestas de los ítems elegidos en una matriz completa ya existente.

En las tablas publicadas, ATLAS usa de media 30–89 ítems. En WinoGrande, TruthfulQA y HellaSwag obtiene MAE de habilidad de 0,155/0,166/0,179, 0,064/0,073/0,071 y 0,157/0,163/0,165 para los tres umbrales. En GSM8K no supera en MAE a MetaBench-Secondary: ATLAS queda en 0,150–0,177 frente a 0,096; en ARC el umbral 0,1 sí obtiene el menor MAE, 0,084. El IES favorece sistemáticamente a ATLAS porque multiplica el error relativo por el número de ítems relativo a Random100; puede mejorar aunque aumente el error absoluto. La reconstrucción p-IRT queda por debajo de 5 puntos porcentuales de MAE, pero mezcla respuestas observadas con probabilidades IRT para los ítems no administrados y la compara con accuracy cruda de un banco que, en varios benchmarks, contiene ítems eliminados de la calibración. Los experimentos de generalización solo se realizan en ARC: el holdout de la familia Mixtral mantiene MAE de 0,091–0,109, mientras el corte temporal empeora a 0,126–0,162.

La auditoría del código identifica un problema que afecta al núcleo de la validación. Si la escala enlazada es θref=A·θk+B y mirt usa el predictor a·θ+d, la transformación que conserva la curva es a*=a/A y d*=d−aB/A. El código y el README usan d*=A·d+B·a. Por tanto, los interceptos enlazados no representan los mismos ítems en la escala de referencia. La correlación que el script imprime después del linking no detecta el error porque correlacionar θref con una transformación afín de θk es invariante al desplazamiento y escala. Como el WLE de banco completo, la selección adaptativa, las probabilidades p-IRT y la mayoría de los baselines usan esos parámetros combinados, el fallo contamina el referente contra el que se afirma que ATLAS coincide.

Hay dos errores adicionales de reporte. La Tabla 1 llama RMSEA a valores de TinyBenchmarks entre 364,24 y 646,82, aunque RMSEA no puede tomar esa escala; además, el script liberado para TinyBenchmarks no calcula M2 ni RMSEA. Y la exposición media se calcula solo sobre ítems que aparecieron al menos una vez, omitiendo los ceros pese a que la fórmula del paper promedia sobre todo el banco. Por ejemplo, HellaSwag SE≤0,1 se reporta como 3,86%, pero el promedio correcto sobre 5.600 ítems es 0,73%; 4.542 ítems no se usan y el ítem más expuesto llega a 40,67%. En GSM8K, el máximo llega a 56,43%. La métrica publicada no demuestra utilización amplia ni evita concentración. En consecuencia, ATLAS ofrece una idea prometedora y artefactos más completos de lo habitual, pero sus cifras no validan todavía un sistema listo para reemplazar benchmarks estáticos: requieren corregir el linking, rehacer los resultados y evaluar prospectivamente coste, contenido y validez externa.

Research question

Can a 3PL adaptive test that selects items by Fisher information estimate IRT ability and reconstruct the accuracy of models with many fewer items than a static benchmark, maintaining ranking, precision and transfer to model families or periods not used in calibration?

Method

Retrospective analysis of binary matrices from the Open LLM Leaderboard for five benchmarks. Models and items are filtered, split by models into calibration/test, 3PL models are fitted by blocks and linked with common persons. WLE over the entire filtered bank acts as reference. CAT uses EAP, randomesque top-5, minimum 30, maximum 500 and SE thresholds 0.1/0.2/0.3. MAE of theta, IES and p-IRT reconstruction is compared with Random100, TinyBenchmarks and two MetaBench subsets; split by Mixtral family, temporal split, split-half, Q3, 2PL/3PL, WLE/EAP and MMLU extension are added.

Sample: There are no human participants or new model calls. The units are model entries and checkpoints from the Open LLM Leaderboard, including variants, fine-tunes, quantizations and related models. After filters, 4,680/521 calibration/test entries are used in WinoGrande, 4,635/516 in TruthfulQA, 3,467/386 in HellaSwag, 3,775/420 in GSM8K and 3,747/417 in ARC. The temporal split of ARC uses 2,998 entries before and 322 after 1 May 2024, excluding 844 without date.

Findings

  • The current version is arXiv v3, revised on 13 July 2026 and accepted as Spotlight at ICML 2026.
  • The PDF carries the notice of Proceedings of the 43rd ICML, PMLR 306, 2026.
  • The work is adaptive evaluation of LLM capabilities, not personality measurement; the Personality keyword from the old record was incorrect.
  • ATLAS operates retrospectively on complete responses and does not reduce in this experiment the cost already incurred to obtain them.
  • The five published calibrated banks have 1,045, 627, 5,600, 1,306 and 839 parameters, not all the rounded or inconsistent counts from the paper and README.
  • The raw test matrices contain 1,045, 644, 5,711, 1,306 and 844 items respectively.
  • ATLAS reconstruction averages over the calibrated bank, while actual_accuracy is computed over the raw matrix; the estimands differ for TruthfulQA, HellaSwag and ARC.
  • In WinoGrande, ATLAS achieves MAE 0.155 with 70 items, 0.166 with 37 and 0.179 with 32.
  • In TruthfulQA it achieves 0.064 with 48 items and approximately 0.07 with the minimum of 30.
  • In HellaSwag it achieves 0.157 with 41 items and 0.163 to 0.165 with 30.
  • In GSM8K, MetaBench-Secondary obtains a lower absolute MAE, 0.096, than the three ATLAS configurations, 0.150 to 0.177.
  • In ARC, ATLAS SE<=0.1 obtains the lowest ability MAE, 0.084 with 89 items; relaxed thresholds increase MAE to 0.120 and 0.117.
  • The IES of ATLAS is the lowest because it combines error and length multiplicatively; it does not always imply the lowest error.
  • p-IRT reconstruction remains below 0.05 MAE in the five tasks, but ATLAS is not always better than static baselines in accuracy.
  • In TruthfulQA, Random100 and MetaBench-P have lower accuracy MAE than ATLAS; in GSM8K the four static alternatives outperform ATLAS.
  • The 23 to 31% of changes of more than ten positions describes difference between rankings, not proof that theta is more correct.
  • The whole-bank theta reference is defined by the same IRT model and parameters used by ATLAS, so the test validates internal approximation, not external validity of capability.
  • Split-half stability of IRT rankings is higher than that of accuracy in the reported results.
  • Correlations between GSM8K and MMLU Mathematics and between two MMLU chemistry subsets increase with theta, but only two related pairs are studied.
  • The Mixtral holdout in ARC maintains MAE 0.091 to 0.109 against 0.084 to 0.117 in random split.
  • The temporal split of ARC worsens MAE to 0.126 to 0.162; accuracy reconstruction changes less, to 0.044 to 0.045.
  • The MMLU extension shows that in some subjects SE<=0.1 uses practically the entire bank, so the saving is not uniform.
  • The script selects the first item randomly within +/-0.5 of difficulty, while the paper's algorithm prescribes the item deterministically closest to theta=0.
  • The Random100 baseline uses a single fixed sample with seed 42, not multiple random subsets.
  • The standard errors in the tables are dispersion of errors between models divided by square root of n, not uncertainty from repetition of selection or calibration.
  • The implemented intercept linking does not preserve the 3PL predictor under the declared scale transformation.
  • The post-link correlation printed by the script is unable to validate the transformed intercept.
  • The linking error affects combined parameters, reference WLE, CAT, p-IRT and derived baselines.
  • Table 1 labels TinyBenchmarks values from 364.24 to 646.82 as RMSEA, an impossible scale for RMSEA.
  • The released TinyBenchmarks script does not run M2 or RMSEA, so those five numbers are not reproducible from the presented code.
  • The RMSEA of ATLAS are averaged across partitions; an acceptable average may hide partitions or dimensions with worse fit.
  • The published exposure average omits all items never selected and contradicts the formula in the appendix.
  • HellaSwag SE<=0.1 is published as 3.86% mean exposure; including the zeros it is 0.73%, with 4,542 of 5,600 items never used.
  • Exposure maxima reach 40.67% in HellaSwag and 56.43% in GSM8K, incompatible with interpreting the published average as absence of concentration.
  • The times of 9 to 76 seconds measure selection and lookup over existing responses, not LLM inference, calibration or end-to-end evaluation.
  • The repository contains 436 files, matrices, parameters, results, scripts and adaptive paths, but does not declare a license.
  • The README says that the matrices are not committed, although data/data.zip contains the 11 calibration and test matrices.
  • There is no public script that derives the raw matrices, documents its exact version of the leaderboard and reproduces all filters and splits from the original source.
  • There is no lockfile, renv, container, automated tests or CI to fix the R/Python environment.
  • The repository allows auditing many intermediate figures, but not reliably reproducing the main conclusion without correcting the code and recalculating the results.

Limitations

  • The evaluation is retrospective; the model already answered all items before simulating the adaptive test.
  • Real cost of calls, tokens, latency, batching, failures or retries of a prospective evaluation is not measured.
  • Selection time is not end-to-end runtime despite the language used in the body of the article.
  • The method presupposes a calibrated bank with thousands of complete historical evaluations.
  • The break-even point between calibration cost and saving per new model is not quantified.
  • The calibration and test models come from the same leaderboard and harness ecosystem.
  • The rows are not independent units: they include related families, checkpoints, fine-tunes, merges and quantizations.
  • Dependence between model-persons can bias parameters and uncertainty and is not studied.
  • Q3 examines local dependence between items, not genealogical dependence between models.
  • Retaining only models with complete response introduces selection by availability.
  • Removing the lower tail but keeping the upper tail asymmetrically modifies the ability distribution.
  • The stratified split shares distribution and may overestimate ordinary transfer.
  • Transfer by architecture and date is only tested in ARC.
  • Only one family, Mixtral, is held out; there is no replication with Llama, Qwen, Gemma or other families.
  • The temporal split excludes 844 entries without date and does not analyze the bias of that exclusion.
  • Uncertainty intervals for the difference between random, family and temporal splits are not published.
  • There is no prospective validation with truly new models after freezing parameters.
  • The unidimensional 3PL model may be inadequate for benchmarks that mix subskills.
  • Dimensionality is not evaluated before interpreting a single theta as capability.
  • M2 is computed per partition, not as a joint fit of the entire linked bank.
  • Averaging RMSEA across partitions does not guarantee global fit or parameter invariance.
  • CFI and TLI of several partitions are low, especially in WinoGrande and ARC, although the narrative highlights RMSEA.
  • Table 1 of TinyBenchmarks contains impossible values under the RMSEA label.
  • The TinyBenchmarks pipeline does not publish the calculation that generated Table 1.
  • The claim of severe misfit of TinyBenchmarks is not verifiable with the released artifact.
  • Filtering by point-biserial correlation does not document whether it uses corrected item-total correlation; the preprocessing pipeline is not published.
  • If the total score includes the item, the correlation is inflated by part-whole overlap.
  • Removing negative correlations can delete subskills or problematic keys without content audit.
  • Accuracy above 95% is treated as uninformative, but coverage of essential easy capabilities is not analyzed.
  • There is no manual review of items discarded due to error, ambiguity or content.
  • The 3PL pseudo-guessing parameter has no unique interpretation for LLM and can capture priors, format, memorization or heuristics.
  • The 2PL/3PL sensitivity shows that 3PL does not uniformly improve MAE.
  • Uncertainty of item parameters is not propagated to the SE used to stop the CAT.
  • Linking uncertainty is not propagated to the SE or to MAE.
  • There is no bootstrap by calibration, item, model family or adaptive selection.
  • The arbitrary minimum of 30 items dominates many conditions and makes SE<=0.2 and SE<=0.3 coincide.
  • The maximum 500 can truncate extreme models without separate diagnosis.
  • The EAP prior and its parameters do not receive a detailed sensitivity analysis.
  • The WLE/EAP comparison in the appendix does not correct the previous linking error.
  • The first item in the code does not match the algorithm described in the paper.
  • Randomesque uses a single seed per model and does not report variation across repetitions.
  • Random100 is a single fixed sample, so it does not represent the distribution of random baselines.
  • The tabulated SEs across models are not repeatability intervals of the method.
  • The baselines use different sizes and, for TinyBenchmarks, recalibrated parameters or banks with different counts.
  • The n_all_items of baselines vary with respect to the ATLAS bank, reducing comparability of IES and p-IRT.
  • The d linking formula in code and README is incompatible with the parameterization a*theta+d and theta_ref=A*theta_k+B.
  • Validation by correlation after linking is tautological with respect to an affine transformation and does not prove equivalence of curves.
  • The first block is chosen as an arbitrary reference without sensitivity analysis to the reference block.
  • Common-item anchors are not used; stability depends on common model-persons and on their scores per block being comparable.
  • The whole-bank WLE called ground truth is an estimate dependent on the same model, filters and linking.
  • The CAT comparison against that WLE is circular internal validation, not external truth of capability.
  • Rank shifts with respect to accuracy are expected when IRT reweights difficulty and discrimination.
  • There is no external criterion that determines whether ranking changes are improvements or distortions.
  • Correlations between related benchmarks cover only mathematics and two chemistry subjects.
  • There are no formal tests or intervals for the correlation differences between accuracy and theta.
  • Split-half ranking stability measures relative reliability, not construct validity.
  • p-IRT reconstructs unobserved responses with the same model that defines theta and can inherit its misfit.
  • The p-IRT comparison uses raw accuracy from matrices that include items outside the filtered bank.
  • Probabilistic calibration of per-item predictions is not reported.
  • Aggregate MAE can hide systematic error in high- or low-ability models.
  • The identity figure does not replace analysis of bias, coverage or limits of agreement.
  • IES is a new and arbitrary index that multiplies two ratios without external validation.
  • IES can decrease when using fewer items even if MAE worsens materially.
  • The strongest accuracy-efficiency tradeoff label depends entirely on that function and on the single Random100.
  • The mean exposure defined over the entire bank is algebraically mean length divided by bank size.
  • The code computes the mean only among items observed at least once and therefore does not implement the published formula.
  • The exposure average does not report the maximum, tail or real concentration.
  • The reconstructed maxima exceed 40% in the five tasks and reach 56.43% in GSM8K.
  • Low test overlap does not demonstrate thematic coverage or absence of content bias.
  • No content, domain, secondary difficulty, format or fairness constraints are imposed in the experiments.
  • The discussion says the framework supports content balancing, but does not implement or evaluate it.
  • Adaptive reduction can concentrate on informative subdomains and lose content validity.
  • Low exposure during CAT does not reduce training contamination of public items already known by the models.
  • Secret banks, temporal rotation or item compromise are not evaluated.
  • The version, snapshot, date and harness of the Open LLM Leaderboard used are not precisely documented.
  • There is no data card with provenance, licenses and transformation of each benchmark.
  • Code to reconstruct the matrices from original leaderboard results is not published.
  • The ZIP contains data that the README declares absent, a sign of desynchronized documentation.
  • The number of items differs between paper, README, raw matrices, parameters and summaries.
  • There is no repository license to reuse code, parameters or matrices.
  • There are no fixed versions of mirt, catR, R, pandas or numpy.
  • The scripts install packages at runtime and do not provide an immutable environment.
  • There are no unit tests for 3PL probability, linking, mapping, stopping, exposure or IES.
  • There is no CI that runs the pipeline or checks the tables.
  • There is no regression test that would have detected the impossible RMSEA or the mis-averaged exposure.
  • Full execution requires manually decompressing data and selected_items, in addition to considerable R resources.
  • The result files derive from multiple scripts and inconsistent naming conventions.
  • The summary script subtracts vectors by position, it does not join by model id, although the main audited files preserve the same order.
  • NA handling in 02_wle_scoring creates data_clean but then operates again on data, nullifying that cleaning.
  • The remaining non-convergent WLE rows are replaced by the median without a flag per model in the output.
  • It is not quantified how many median imputations occurred in the published results.
  • The MMLU extension does not present the same complete set of baselines, fit, Q3 and generalization splits.
  • In some MMLU subjects the strict threshold uses almost or exactly all items.
  • The claim of approximately 80% reduction in MMLU refers to the relaxed threshold and not to all settings.
  • There is no evaluation of generative questions, non-binary metrics, LLM judges or open tasks.
  • There is no evaluation of safety, bias, multilingualism or prompt variation.
  • It is not demonstrated that theta is comparable across different benchmarks; each bank has its own scale.
  • Parameter drift when models, harnesses or bank contamination change is not analyzed.
  • There is no empirical strategy for online recalibration or detection of loss of invariance.

What the study does not establish

  • It does not establish that ATLAS can replace static benchmarks without correcting and recalculating the linking.
  • It does not demonstrate a real end-to-end cost reduction because the simulation uses already available responses.
  • It does not demonstrate that the published times include LLM inference.
  • It does not establish that theta is ground truth of general capability.
  • It does not demonstrate that rank shifts are more valid than ranking by accuracy.
  • It does not demonstrate the lowest absolute MAE in all benchmarks or metrics.
  • It does not demonstrate that ATLAS outperforms MetaBench-Secondary in GSM8K.
  • It does not demonstrate that reconstructed accuracy uses exactly the same bank as raw accuracy in all tasks.
  • It does not demonstrate good fit of TinyBenchmarks with a valid or reproducible RMSEA comparison.
  • It does not demonstrate broad item utilization with the implemented exposure average.
  • It does not demonstrate low maximum exposure; some items appear in more than half of the tests.
  • It does not demonstrate thematic coverage, fairness or content balancing.
  • It does not demonstrate lower pretraining contamination on public benchmarks.
  • It does not demonstrate generalization to all families, dates, tasks or leaderboards.
  • It does not demonstrate external validity with independent outcomes from the same benchmarks.
  • It does not allow reproducing from the original source the complete construction of the matrices and splits.
  • It does not currently offer a licensed artifact with a fixed environment ready for production adoption.

Traceability

Scope: Full text

Version: arXiv:2511.04689v3, revised 13 July 2026; ICML 2026 Spotlight with PMLR 306 proceedings notice; 24 pages; official repository commit c0bc19b33aaa11703b8258377abb367533b8a824 audited as supplement

Consulted source: https://arxiv.org/pdf/2511.04689v3

Review: Codex ICML-version reconciliation, complete bilingual full-text fidelity pass, all-page PDF visual inspection, table and formula audit, independent metric recomputation, raw-versus-filtered bank reconciliation, construct and generalization audit, and official-repository code/data audit at pinned commit; summaries written from the complete paper and artifacts rather than abstract keywords, 2026-07-15

Approval: Codex fidelity pass, 2026-07-15

English translation: approved, 2026-07-18

Models evaluated

  • 4,680 calibration and 521 test leaderboard model entries for WinoGrande
  • 4,635 calibration and 516 test leaderboard model entries for TruthfulQA
  • 3,467 calibration and 386 test leaderboard model entries for HellaSwag
  • 3,775 calibration and 420 test leaderboard model entries for GSM8K
  • 3,747 calibration and 417 test leaderboard model entries for ARC
  • 321 Mixtral-family ARC entries used only in strict architecture-family holdout testing
  • 322 post-2024-05-01 ARC entries used in temporal testing after excluding 844 entries without release dates
  • No LLM is queried prospectively by the released ATLAS experiments

Instruments and metrics

  • Three-parameter logistic Item Response Theory model in mirt parameterization
  • Partitioned calibration with common-person mean–sigma linking
  • Whole-filtered-bank WLE reference ability
  • EAP adaptive ability updates
  • Fisher-information item ranking with random sampling from the top five
  • SE stopping thresholds 0.1, 0.2 and 0.3, minimum 30 and maximum 500 items
  • Ability MAE and standard error across model-level absolute errors
  • p-IRT reconstructed accuracy from observed items and modeled unobserved probabilities
  • Information Efficiency Score relative to a single Random100 subset
  • Limited-information M2, RMSEA and 2PL Q3 local-dependence diagnostics
  • Split-half Spearman ranking stability and related-benchmark rank correlation
  • Average item exposure, test overlap and adaptive-selection-loop time

Data used

  • Open LLM Leaderboard-derived WinoGrande response matrix: 5,201 model rows before 90/10 split; 1,045 calibrated items after filters
  • Open LLM Leaderboard-derived TruthfulQA response matrix: 5,151 model rows; 627 calibrated items after filters
  • Open LLM Leaderboard-derived HellaSwag response matrix: 3,853 model rows; 5,600 calibrated items after filters
  • Open LLM Leaderboard-derived GSM8K response matrix: 4,195 model rows; 1,306 calibrated items after filters
  • Open LLM Leaderboard-derived ARC response matrix: 4,164 model rows; 839 calibrated items in released parameter bank
  • Five released test matrices contain 521, 516, 386, 420 and 417 complete model response rows
  • MMLU per-subject appendix extension without the same full baseline and diagnostic reporting
  • Official repository data/data.zip with 11 CSVs totaling approximately 87 MB
  • Official repository selected_items.zip archives with per-model adaptive paths for all five benchmarks

Evidence and location

  • Version v3, authorship, revision of 13 July 2026 and ICML 2026 Spotlight: arXiv:2511.04689v3 metadata and PDF front matter, page 1
  • Model selection, split, filters, partitions and CAT: Paper Sections 3.1 to 3.4, Algorithm 1 and Appendix C
  • Main results of ability, IES and accuracy: Paper Tables 2, 3, 9 and 10; repository summary_theta_mae_sd_se.csv and summary_pirt_mae_sd_se.csv
  • Split-half, related benchmarks and rank shifts: Paper Sections 4.4 to 4.5 and Table 4
  • Family and temporal generalization: Paper Section 4.6 and Table 5; repository experiments/arc_by_family_mixtral
  • MMLU extension and 2PL/3PL sensitivity: Paper Appendix G.3 to G.5, Tables 12 to 14
  • Incorrect d transformation in mean-sigma linking: Official repository commit c0bc19b: scripts/02_wle_scoring.r lines 69 to 78 and README IRT Model equations; compared with paper's mirt predictor a*theta+d
  • Impossible TinyBenchmarks RMSEA and missing released computation: Paper Table 1; repository experiments/baseline_compare/irt_tinybenchmark_items.r, which fits 3PL but never calls M2 or RMSEA
  • Exposure implementation omits never-selected items: Paper Appendix E equations 5 to 6 and Table 11; repository scripts/03_atlas_cat.r lines 454 to 504 and scripts/analysis/calculate_item_overlap.py lines 105 to 132
  • Recomputed exposure, unused items and maxima: All five repository selected_items.zip archives at commit c0bc19b; independent aggregation over the complete published parameter-bank sizes
  • Single Random100 sample: Repository experiments/baseline_compare/random_100.r lines 65 to 66 and one selected-items CSV per benchmark
  • Retrospective score lookup and selection-only timing: Repository scripts/03_atlas_cat.r item administration and time measurement; paper Appendix G.2
  • Raw versus calibrated bank count mismatch: Repository data/data.zip CSV headers, irt_item_parameters_combined.csv row counts, scripts/04_pirt_accuracy.r and scripts/05_compute_actual_acc.r
  • Artifact coverage, missing preprocessing provenance, environment and license: Complete file tree and git metadata of official repository commit c0bc19b; 436 tracked files
  • Visual inspection: All 24 pages of arXiv:2511.04689v3 rendered and visually inspected on 15 July 2026