{"id":"paper-autoraters","name":"paper-autoraters","summary":"PaperOrchestra(arXiv:2604.05018、付録F.3)の4つの論文品質自動評価ツールを実行します — 引用F1(P0/P1分割+精度/リコール/F1)、文献レビュー品質(6軸0-100、反インフレルール付き)、SxS総合論文品質(並列)、およびSxS文献レビュー品質(並列...)","body":"# Paper Autoraters (App. F.3)\n\nFaithful implementation of the four LLM-as-judge autoraters used in\nPaperOrchestra (Song et al., 2026, arXiv:2604.05018, §5 and App. F.3).\n\nThese are the metrics the paper uses to demonstrate that PaperOrchestra\nbeats single-agent and AI-Scientist-v2 baselines. Use them to:\n\n1. Score a generated paper against a ground-truth paper.\n2. Compare two paper-writing pipelines side-by-side.\n3. Validate your own host-agent execution of the paper-orchestra pipeline.\n\n## The four autoraters\n\n| Autorater | What it does | Inputs | Output |\n|---|---|---|---|\n| **Citation F1 — P0/P1 partition** | Partitions reference list into P0 (must-cite) and P1 (good-to-cite) given the paper text | one paper text + its references list | JSON `{ref_num: \"P0\"\\|\"P1\"}` |\n| **Literature Review Quality** | 6-axis 0-100 score for Intro+Related Work, with anti-inflation hard caps | one paper PDF/text + reference avg citation count | JSON with `axis_scores`, `penalties`, `summary`, `overall_score` |\n| **SxS Overall Paper Quality** | Holistic side-by-side preference judgment | two papers (PDF or text) | JSON with `winner` ∈ {paper_1, paper_2, tie} |\n| **SxS Literature Review Quality** | Side-by-side preference, Intro+Related Work only | two papers | JSON with `winner` ∈ {paper_1, paper_2, tie} |\n\nThe paper uses Gemini-3.1-Pro and GPT-5 as judges, set to temperature 0.0\n(Gemini) or default 1.0 (GPT-5, which doesn't allow temperature\nadjustment). Use whatever your host LLM is.\n\n## Workflow\n\n### Citation F1 (compute Precision / Recall / F1 vs ground truth)\n\nThis is a two-step procedure:\n\n#### Step 1: Partition the reference lists into P0 / P1\n\nFor both the ground-truth paper AND the generated paper, run the LLM with\n`references/citation-f1-prompt.md`:\n\n```\ninputs:\n  paper_text:    full paper LaTeX or markdown\n  references_str: numbered reference list (e.g., \"1. Vaswani et al. (2017)\n                  Attention Is All You Need. NeurIPS. 2. He et al. (2016)\n                  Deep Residual Learning for Image Recognition. CVPR. ...\")\n\noutput: JSON {\"1\": \"P0\", \"2\": \"P1\", \"3\": \"P0\", ...}\n```\n\nSave both partitions:\n- `bench/<paper_id>/gt_partition.json`\n- `bench/<paper_id>/gen_partition.json`\n\n#### Step 2: Resolve references to entity IDs and compute F1\n\nThe paper uses Semantic Scholar paper IDs to match references between the\ntwo lists. The `compute_f1.py` script does this deterministically given\ntwo input lists:\n\n```bash\npython skills/paper-autoraters/scripts/compute_f1.py \\\n    --gt-partition gt_partition.json \\\n    --gt-refs gt_refs.json \\\n    --gen-partition gen_partition.json \\\n    --gen-refs gen_refs.json \\\n    --out f1_report.json\n```\n\nWhere `gt_refs.json` and `gen_refs.json` are lists of `{ref_num,\npaper_id, title}` produced by your host's S2-resolution pass (the same\nfuzzy match + S2 verification used by `literature-review-agent/scripts/`).\n\nOutput JSON contains P0 / P1 / overall Precision, Recall, F1.\n\n### Literature Review Quality (single paper, 6 axes)\n\nLoad `references/litreview-quality-prompt.md`. Inputs:\n\n- The full paper PDF (or LaTeX/markdown if your host lacks PDF input)\n- `avg_citation_count` for the venue/field (used as the baseline for\n  citation count anchoring, e.g., 58.52 for CVPR 2025, 59.18 for ICLR 2025\n  per the paper)\n\nThe prompt instructs the model to evaluate ONLY the literature-review\nfunction of the paper (Introduction + Related Work / Background sections).\nIt produces a strict JSON output with per-axis scores and justifications.\n\nCritical anti-inflation rules baked into the prompt:\n\n| Rule | Cap |\n|---|---|\n| Default expectation | overall 45-70 |\n| > 85 requires strong evidence on ALL axes | — |\n| > 90 extremely rare (near-survey-level mastery) | — |\n| Any axis < 50 → overall rarely > 75 | — |\n| Mostly descriptive review | Critical Analysis ≤ 60 |\n| Novelty asserted without comparison | Positioning ≤ 60 |\n| Sparse/inconsistent citations | Citation Rigor ≤ 60 |\n| Citation count < 50% of avg | Coverage ≤ 55 |\n| Citation count > 120% of avg | Coverage = \"strong\" |\n\nPlus penalty table:\n\n| Penalty | Range |\n|---|---|\n| Overclaiming novelty | -5 to -15 |\n| Missing key recent work | -5 to -15 |\n| Mostly descriptive review | -5 to -10 |\n| Weak gap statements | -5 to -10 |\n| Citation dumping | -5 to -10 |\n\nSave the output to `litreview_quality_score.json`. The score JSON is the\nsame shape used by `content-refinement-agent/scripts/score_delta.py`, so\nyou can re-use the halt-rule logic to compare iterations.\n\n### SxS Overall Paper Quality (side-by-side, full paper)\n\nLoad `references/sxs-paper-quality-prompt.md`. Inputs:\n\n- Two paper PDFs or LaTeX files (call them `paper_1` and `paper_2`)\n\nThe prompt produces a JSON with `paper_1_holistic_analysis`,\n`paper_2_holistic_analysis`, `comparison_justification`, and\n`winner ∈ {paper_1, paper_2, tie}`.\n\nTo mitigate LLM positional bias (the paper notes this in §5.4), run the\ncomparison **twice** with the order swapped:\n\n```\ncall_1: paper_A → paper_1, paper_B → paper_2  → winner1\ncall_2: paper_B → paper_1, paper_A → paper_2  → winner2\n```\n\nFinal outcome: a `win` (both calls agree on paper A), `tie` (one win + one\ntie, or two ties), or `loss` (both agree on paper B). The paper uses this\nexact ordering protocol.\n\n### SxS Literature Review Quality (side-by-side, Intro+RW only)\n\nLoad `references/sxs-litreview-prompt.md`. Same input/output shape as the\nSxS paper quality autorater, but the model is instructed to evaluate\n**only** the Introduction and Related Work / Background sections of each\npaper. Same positional-bias mitigation: run twice, swap order.\n\n## Resources\n\n- `references/citation-f1-prompt.md`        — verbatim P0/P1 partition prompt from App. F.3\n- `references/litreview-quality-prompt.md`  — verbatim 6-axis litreview rubric from App. F.3\n- `references/sxs-paper-quality-prompt.md`  — verbatim SxS paper-quality prompt from App. F.3\n- `references/sxs-litreview-prompt.md`      — verbatim SxS litreview prompt from App. F.3\n- `scripts/compute_f1.py` — Precision / Recall / F1 from two partition JSONs","author":"@Ar9av","ownerProfile":null,"authorContacts":null,"sourceUrl":"https://github.com/Ar9av/PaperOrchestra/tree/main/skills/paper-autoraters","license":"MIT","category":"writing","lang":"en","tokens":1646,"stars":0,"calls30d":2,"claimed":false,"visibility":"public","origin":"crawler","version":"0.1.0","createdAt":"2026-08-22","updatedAt":"2026-08-22","files":[{"path":"references/citation-f1-prompt.md","size":2563,"sha256":"f91252b5b23a6b7a44c299b9424a55379cbf0ff5bfed592efa7c88c711aff1d9"},{"path":"references/litreview-quality-prompt.md","size":8060,"sha256":"6d0a9e41eb2111766838cfdaa940107dc789e78fbff2c9d87da2df5a7d06f93b"},{"path":"references/sxs-litreview-prompt.md","size":2406,"sha256":"222ee853767444dcb24e9008d877d97832a015e577db1eafaf98ebe9f7e0d9ed"},{"path":"references/sxs-paper-quality-prompt.md","size":3433,"sha256":"8c6252be2ec847c9443858c1ffed8739ca02f4db216bfabb9ef4ffbcdb79702b"},{"path":"scripts/compute_f1.py","size":3690,"sha256":"3a707aa0ba60dfbff4c5cf4328ba0c71f9dd244bab0a94fe2b30664d0514edee"}],"requires":{"mcp":[],"tools":[]},"safety":{"flags":[],"scannedAt":"2026-08-22","hasScripts":true,"networkEndpoints":[]}}