Ranking Agreement Does Not Reveal Paragraph-Edit Agreement in Two Released Paper Acceptance Predictors
Organizations: Department of Mechanical Engineering, Santa Clara University 500 El Camino Real, Santa Clara, CA 95053, USA
Abstract
A fast acceptance classifier could select revisions for a slower evaluator. We ask whether agreement on existing papers supports that use. We compare a classifier of word and document features with two released text-only acceptance predictors fine-tuned from the same language model with three billion parameters. For 198 papers from the International Conference on Learning Representations, we compare model rankings of the original papers and of paragraph rewrites screened automatically for unchanged facts, claim strength, and attribution. The classifier ranks original papers closely with one predictor and moderately with the other, but shows no detectable agreement with either on which rewrites raise scores. The tests specified in advance used reduced numerical precision. Rounding error dominated the small score responses. We therefore rescored the released weights in full 32-bit precision as a post hoc check; edit agreement remained near zero. At this precision, when each predictor chooses among five rewrites per paper, its average raw score is under two points out of 100 higher than with random choice. The evidence covers presentation rewrites and two open predictors from one family; transfer to costly evaluators remains untested. Revision selectors should therefore be checked against the intended evaluator on the revisions they rank.
Figures & tables
| Evaluator | AUC [95% CI] | Spearman with | Time per paper |
|---|---|---|---|
| 1.000 | 6.98 ms | ||
| 0.731 | 1.406 s | ||
| 0.547 | 1.405 s |
| Purpose | Forward calculations | Analysis status |
|---|---|---|
| Planned tests | bfloat16, Transformers | Four pre-specified tests |
| Released-weight check | 32-bit, Transformers | Post hoc; basis for interpreting the released weights |
| Release-recipe proxy | float16, Transformers | Post hoc approximation |
| Release recipe | float16, vLLM default | Not run |
| Set | Papers | Inputs or role |
|---|---|---|
| Study 2 selected | 200 | Starting originals for single edits |
| With admitted edits | 199 | 3,323 single-edit versions |
| Edit-ranking analysis | 198 | 3,321 versions meeting the eligibility rule |
| Study 3 search | 128 | Same papers under adaptive and random search |
| Standardization | 328 | The 200 Study 2 and 128 Study 3 originals combined |
| Test | Target | Papers | Estimate [95% CI] | Adjusted |
| Edit fidelity | 198 | 1.0 | ||
| 198 | 1.0 | |||
| Search contrast | 128 | 1.0 | ||
| 128 | 1.0 | |||
| Target agreement | vs | 198 | – | |
| Targets rescored in 32-bit floating point (post hoc; estimates for the released weights) | ||||
| Study | Outcome examined | Relation to this study |
|---|---|---|
| PaperScorer ( Konan et al., 2026c ) | Own-model feedback on section rewrites of 30 borderline papers | We select revisions with a separate classifier. |
| Yang et al. (2026b) | Across-paper Pearson correlations: 0.804 on original scores; 0.085–0.415 on revision gains | We rank several paragraph alternatives within each paper. |
| RobustReview ( Wang et al., 2026 ) | Stability under preserving rewrites and discrimination across papers | These are complementary to agreement between scorers on edits. |
| This study | Within-paper edit rankings, search, and choices from equal-size banks | Edits are fixed before target scoring; preservation errors limit interpretation. |
| Study | Item | Count | Invariants pass | Both pass |
|---|---|---|---|---|
| 2 | Presentation edit | 4,712 | 90.9% | 70.5% |
| 2 | Evidence removal | 198 | 98.5% | 51.5% |
| 2 | Deletion sham | 198 | 98.5% | 92.9% |
| 3 | Presentation edit | 3,002 | 92.5% | 73.7% |
| 3 | Polish paragraph | 4,616 | 92.9% | 74.8% |
| Policy | Budget | Nonempty endpoints | Pass | Fail |
|---|---|---|---|---|
| Adaptive | 64 | 254 | 242 (95.3%) | 12 |
| Adaptive | 512 | 254 | 242 (95.3%) | 12 |
| Random | 64 | 255 | 247 (96.9%) | 8 |
| Random | 512 | 256 | 246 (96.1%) | 10 |
| Item | Audited | Invalid | Invalid rate [95% CI] | One-sided upper bound |
|---|---|---|---|---|
| Composite endpoint | 59 | 0 | 0.0% [0.0, 6.1] | 4.95% |
| Adaptive-labeled subset | 19 | 0 | 0.0% [0.0, 17.6] | 14.6% |
| Presentation edit | 20 | 0 | 0.0% [0.0, 16.8] | 13.9% |
| Evidence removal | 20 | 6 | 30.0% [11.9, 54.3] | 50.8% |
| Type | Papers | Edits | ||
|---|---|---|---|---|
| Clarity | 199 | 1566 | ||
| Emphasis | 115 | 167 | ||
| Flow | 197 | 889 | ||
| Framing | 84 | 129 | ||
| Structure | 193 | 572 |
| Budget | |||
|---|---|---|---|
| 1 | |||
| 8 | |||
| 64 | |||
| 512 |
| Choice or contrast | ||
|---|---|---|
| Uniform random expectation | ||
| Surrogate choice | ||
| Sibling choice | ||
| Target oracle | ||
| Surrogate minus random | ||
| Sibling minus random |
| Quantity | |||
|---|---|---|---|
| Absolute single-edit change | |||
| Adaptive minus original | |||
| Random minus original | |||
| Polish minus original |
| Subset | Papers | Edits | [95% CI] | [95% CI] |
|---|---|---|---|---|
| All eligible edits | 198 | 3321 | ||
| Common-visible spans | 198 | 3072 | ||
| Complete documents fit | 6 | 120 |
| Evaluator | Mean [95% CI] | Win | Tie | Loss |
|---|---|---|---|---|
| 108 | 7 | 13 | ||
| 59 | 7 | 62 | ||
| 58 | 7 | 63 |
| Policy | Papers | Median | Interquartile range | Range |
|---|---|---|---|---|
| Adaptive | 128 | 93 | 75–120.25 | 15–156 |
| Random | 128 | 512 | 512–512 | 512–512 |
| Chooser | advantage [95% CI] | advantage [95% CI] |
|---|---|---|
| Surrogate choice | ||
| Sibling choice | ||
| Target oracle |
| Bank size | Chooser | [95% CI] | [95% CI] |
|---|---|---|---|
| 5 | Surrogate | ||
| 5 | Sibling | ||
| 5 | Target oracle | ||
| 10 | Surrogate | ||
| 10 | Sibling | ||
| 10 | Target oracle |
| Quantity | Scoring | ||
|---|---|---|---|
| Original scores, with RTX 4090 | bfloat16 | 0.9985 | 0.9993 |
| Edit changes vs. RTX 4090 | bfloat16 | ||
| 32-bit | |||
| Edit changes vs. 4090 32-bit | 32-bit | ||
| Mean (SD), real / null | bfloat16 | 0.078 / 0.052 | 0.055 / 0.034 |
| 32-bit (4090) | 0.055 / 0.026 | 0.030 / 0.015 |
| Quantity | Pair | Estimate [95% CI] |
|---|---|---|
| Edit agreement, RTX 4090 bfloat16 | – | |
| Edit agreement, RTX 5090 bfloat16 | – | |
| Edit agreement, RTX 5090 32-bit | – | |
| Whitespace-edit agreement, RTX 5090 bfloat16 | – | |
| Pooled edit changes | – | |
| – |
| Contrast | Paired papers | Mean change [95% CI] |
|---|---|---|
| Adaptive minus original | 19 | |
| Random minus original | 21 | |
| Polish minus original | 24 | |
| Adaptive minus random | 19 |
| Set | Papers | Inputs or inclusion rule |
|---|---|---|
| Study 2 selected | 200 | All retained as originals |
| Admitted single-edit pool | 199 | 3,323 edited versions |
| Edit fidelity | 198 | 3,321 edits; at least five per paper and varying scores |
| Visible-span fidelity | 198 | 3,072 edits; complete changed paragraphs visible to all scorers |
| Paired deletion diagnostic | 93 | One admitted removal and sham per paper |
| Study 3 search | 128 | Same papers under both policies and seeds |
| Component | Identifier and version |
|---|---|
| skonan/PaperScorer-3B-Text-OpenReview-ICLR , revision cc25000d6751 | |
| skonan/PaperScorer-3B-Text-arXiv , revision 5a25bebf9dcc | |
| Source text | skonan/PaperScorer-Text , revision c39ba04eb7d1 |
| Target inference | PyTorch ( Paszke et al., 2019 ) 2.11.0 (CUDA 12.8, NVIDIA’s GPU computing platform), Transformers ( Wolf et al., 2020 ) 5.21.0, Tokenizers 0.23.3; bfloat16 forward pass, 32-bit Accept/Reject logits |
| scikit-learn ( Pedregosa et al., 2011 ) 1.9.1 | |
| DeepReviewer-14B | WestlakeNLP/DeepReviewer-14B ; converted to the GGUF model-file format with Q8_0 (8-bit) quantization; served by llama.cpp release b11972, with a converter patch mapping the model’s tokenizer to llama.cpp’s cl100k-style ( dbrx ) pre-tokenizer |
Automated Evaluation
Automated evaluation develops metrics, benchmarks, and AI judges to assess model outputs and system behavior at scale.
See all 421 papers