{"id":"content-refinement-agent","name":"content-refinement-agent","summary":"PaperOrchestraパイプラインのステップ5(arXiv:2604.05018)。","body":"# Content Refinement Agent (Step 5)\n\nFaithful implementation of the Content Refinement Agent from PaperOrchestra\n(Song et al., 2026, arXiv:2604.05018, §4 Step 5, App. F.1 pp. 49–51).\n\n**Cost: ~5–7 LLM calls** (App. B), typically ~3 refinement iterations, each\nconsisting of one reviewer call and one revision call.\n\nThe paper highlights this step as one of the largest contributors to overall\nquality: refinement alone accounts for +19% (CVPR) and +22% (ICLR) absolute\nacceptance-rate improvement (Fig. 4). Get this step right.\n\n## Inputs\n\n- `workspace/drafts/paper.tex` — output of Step 4\n- `workspace/inputs/conference_guidelines.md`\n- `workspace/inputs/experimental_log.md` — used as ground truth for the\n  hallucination check\n- `workspace/citation_pool.json` / `workspace/refs.bib` — the allowed\n  bibliography\n\n## Outputs\n\n- `workspace/refinement/iter1/`, `iter2/`, `iter3/` — per-iteration snapshots\n  containing `paper.tex`, `paper.pdf`, `review.json`, `score.json`\n- `workspace/refinement/worklog.json` — append-only history of decisions\n- `workspace/final/paper.tex` and `workspace/final/paper.pdf` — copy of the\n  best accepted snapshot\n\n## The refinement loop\n\n```\nprev_score = score(paper.tex)                  # baseline from initial draft\nsnapshot iter0/\n\nfor iter in 1..ITER_CAP (default 3):\n    1. simulate_review(paper.tex) → review.json\n       (uses `references/reviewer-rubric.md` rubric)\n\n    2. apply_revision(paper.tex, review.json) → new_paper.tex\n       (uses verbatim Refinement Agent prompt at `references/prompt.md`)\n\n    3. snapshot iter<N>/ with new_paper.tex, review.json\n       latexmk -pdf new_paper.tex → iter<N>/paper.pdf\n\n    4. score(new_paper.tex) → curr_score\n\n    5. decide via score_delta.py:\n       - if curr.overall > prev.overall:                       ACCEPT\n       - elif curr.overall == prev.overall and net_subaxis ≥0: ACCEPT\n       - else:                                                 REVERT\n\n    6. apply_worklog.py to append the decision\n\n    7. if REVERT or no actionable weaknesses or iter == ITER_CAP: HALT\n\n    paper.tex ← new_paper.tex   (only on ACCEPT)\n    prev_score ← curr_score\n\ncp <best iter>/paper.tex → workspace/final/paper.tex\n```\n\nThe \"best\" snapshot at HALT is the one with the highest accepted overall\nscore. On a REVERT halt, the best is the iteration immediately before the\nrevert.\n\n## Step-by-step\n\n### 0. Pre-refinement integrity gate\n\nBefore snapshotting or scoring the initial draft, run two gates in order:\n\n**Gate A — AI failure modes** (load `references/ai-failure-modes.md`, runs once):\n\nLoad `references/ai-failure-modes.md` (which points to `skills/shared/ai_failure_modes.md`).\nRun all 7 checks against the draft and the inputs. This gate runs **once only**,\nat the start of iteration 1.\n\n- CONFIRMED failure → write HALT entry to worklog.json, report to user, stop.\n- SUSPECTED failure → add WARNING comment to paper.tex, log in worklog.json, continue.\n- No failures → proceed.\n\n**Gate B — Claim-evidence provenance** (runs once, WARN gate):\n\n```bash\npython skills/paper-orchestra/scripts/claim_evidence_gate.py \\\n    --paper workspace/drafts/paper.tex \\\n    --log   workspace/inputs/experimental_log.md \\\n    --out   workspace/claim_evidence_report.json\n```\n\nExit 0 → PASS, proceed normally.\nExit 1 → WARN: unsupported numeric claims found. Log in worklog.json as:\n`{gate: \"claim_evidence\", status: \"WARN\", unsupported_count: N, report: \"workspace/claim_evidence_report.json\"}`\nPass the `unsupported` list from the report to the revision agent in Step 3 as\nan additional instruction: \"The following numeric values appear in the paper but\ncannot be corroborated in experimental_log.md — verify or remove them: ...\"\nDo NOT halt on Gate B warnings; the revision agent will address them.\n\n**Gate C — Read research brief** (every run, no exit code):\n\nIf `workspace/research_brief.md` exists, read it before all reviewer calls.\nPass the \"Sections where evidence was thin\" list from §4 as additional\ncontext to the Devil's Advocate reviewer. This surfaces the highest-risk\nsections for CRITICAL scrutiny.\n\n### 0b. Snapshot the initial draft\n\n```bash\npython skills/content-refinement-agent/scripts/snapshot.py \\\n    --src workspace/drafts/paper.tex \\\n    --dst workspace/refinement/iter0/\n```\n\nThis creates `iter0/paper.tex`. Then compile to `iter0/paper.pdf`:\n\n\n```bash\ncd workspace/refinement/iter0/ && latexmk -pdf -interaction=nonstopmode paper.tex\n```\n\nScore it (see Step 1 below) → `iter0/score.json`.\n\n### 1. Simulate peer review\n\nFor each iteration N starting from 1:\n\n**Writing quality pre-check (start of every iteration):** Load\n`references/writing-quality-check.md` and run the 5-category checklist\n(Categories A–E) against the current draft. Note violations and add them to\nthe revision agenda.\n\n**Update critique memory before the reviewer call** (iter N ≥ 2 only — skip for iter 1):\n\n```bash\npython skills/content-refinement-agent/scripts/update_critique_memory.py \\\n    --worklog workspace/refinement/worklog.json \\\n    --review  workspace/refinement/iter<N-1>/review.json \\\n    --iter    <N> \\\n    --out     workspace/refinement/critique_memory.json\n```\n\nThis produces `critique_memory.json` with `focus_on` (persistent unresolved\nissues) and `do_not_reflag` (already-resolved issues). Inject both lists into\nthe reviewer system prompt verbatim:\n\n```\nCRITIQUE MEMORY — you must honour this before reviewing:\n\nFOCUS ON (flagged in prior iterations, not yet resolved — prioritise these):\n<critique_memory.focus_on items, one per line>\n\nDO NOT RE-FLAG (already addressed in prior iterations):\n<critique_memory.do_not_reflag items, one per line>\n```\n\nThis prevents the reviewer from re-discovering already-fixed issues and\nfrom missing genuinely stuck problems.\n\nLoad `references/reviewer-rubric.md` as the system prompt for the simulated\nreviewer call. The reviewer reads `iter<N-1>/paper.pdf` (or `paper.tex` if\nyour host LLM lacks PDF input) and produces a JSON of strengths,\nweaknesses, questions, and per-axis scores.\n\nThe rubric is structured to mimic AgentReview (Jin et al., 2024) — the\npaper's chosen evaluator. We ship a faithful rubric in the references\ndirectory; the host agent's LLM does the actual reviewing.\n\n**Devil's Advocate reviewer:** One simulated reviewer must be designated the DA\nfollowing `references/da-reviewer.md`. The DA challenges core claims from first\nprinciples (causal overclaiming, ablation coverage, baseline fairness,\ngeneralization claims, novelty inflation) rather than surface polish. If the DA\nissues a CRITICAL finding that remains unaddressed after all reviewers weigh in,\nthat finding blocks the \"refinement accepted\" decision regardless of rubric scores.\nLog DA CRITICAL findings in worklog.json: `{da_critical: true, finding: \"...\"}`.\n\nRecord the DA's per-round findings and concession decisions in\n`workspace/refinement/da_concessions.json` (schema in `references/da-reviewer.md`)\nand enforce the concession-threshold protocol deterministically — this stops the\nsimulated DA from sycophantically caving:\n\n```bash\npython skills/content-refinement-agent/scripts/concession_guard.py \\\n    --log workspace/refinement/da_concessions.json \\\n    --out workspace/refinement/iter<N>/da_guard.json\n# exit 0 = clear; exit 1 = standing CRITICAL → force REVERT this iteration;\n# exit 2 = a concession was rejected (caving/consecutive) → DA must restate;\n# exit 3 = schema error.\n```\n\nThe guard rejects any concession made at `rebuttal_score < 4` or in a round\nimmediately following another concession, and restores the affected finding to\n\"standing\". A standing CRITICAL (exit 1) overrides an ACCEPT into a REVERT.\n\nSave to `workspace/refinement/iter<N>/review.json`.\n\n### 2. Score the draft\n\nThe reviewer call produces both qualitative feedback and a per-axis score:\n\n```json\n{\n  \"axis_scores\": {\n    \"scientific_depth\":     {\"score\": 65, \"justification\": \"...\"},\n    \"technical_execution\":  {\"score\": 70, \"justification\": \"...\"},\n    \"logical_flow\":         {\"score\": 60, \"justification\": \"...\"},\n    \"writing_clarity\":      {\"score\": 55, \"justification\": \"...\"},\n    \"evidence_presentation\":{\"score\": 72, \"justification\": \"...\"},\n    \"academic_style\":       {\"score\": 68, \"justification\": \"...\"}\n  },\n  \"overall_score\": 64.5,\n  \"decision_band\": \"Major Revision\",\n  \"strengths\": [...],\n  \"weaknesses\": [...],\n  \"questions\": [...]\n}\n```\n\nSave to `iter<N>/score.json`. (Combined with `review.json` if your host\nemits one document; the schemas overlap.)\n\n`decision_band` is derived deterministically from `overall_score` — Accept\n(≥80) / Minor Revision (65–79) / Major Revision (50–64) / Reject (<50). Fill it\nin with `python skills/content-refinement-agent/scripts/decision_band.py\n--score-json iter<N>/score.json` rather than by hand, so it can never disagree\nwith the number. The bands drive the target-met halt in Step 5.\n\n### 3. Apply revision\n\nLoad the **verbatim Content Refinement Agent prompt** at `references/prompt.md`.\nPrepend the Anti-Leakage Prompt. Inputs:\n\n- `paper.tex` — current draft\n- `paper.pdf` — compiled PDF (multimodal context if available)\n- `conference_guidelines.md`\n- `experimental_log.md` — ground truth for numeric claims\n- `worklog.json` — history of previous changes\n- `citation_pool.json` — the allowed bibliography\n- `reviewer_feedback` — the JSON from Step 1\n\nThe prompt instructs the model to address weaknesses, integrate question\nanswers, and emit two output blocks:\n\n1. A worklog JSON `{addressed_weaknesses[], integrated_answers[], actions_taken[]}`\n2. The full revised LaTeX code\n\nSave the revised LaTeX as `iter<N>/paper.tex`. Append the worklog JSON to\n`workspace/refinement/worklog.json` via `apply_worklog.py`.\n\n### 4. Compile and re-score\n\n```bash\ncd workspace/refinement/iter<N>/ && latexmk -pdf -interaction=nonstopmode paper.tex\n```\n\nThen re-run the simulated review on the new draft → updated `score.json`\nfor the new iteration. (This is the \"re-score after revision\" call.)\n\n### 5. Apply the accept/revert decision\n\nThe calling loop must track `CONSECUTIVE_SMALL` (starts at 0) and pass it\non each call so `score_delta.py` can detect the plateau:\n\n```bash\npython skills/content-refinement-agent/scripts/score_delta.py \\\n    --prev workspace/refinement/iter<N-1>/score.json \\\n    --curr workspace/refinement/iter<N>/score.json \\\n    --plateau-threshold 1.0 \\\n    --plateau-streak 3 \\\n    --accept-threshold 80 \\\n    --consecutive-small $CONSECUTIVE_SMALL \\\n    > workspace/refinement/iter<N>/delta.json\n\nEXIT=$?\n# Update streak for next iteration:\nCONSECUTIVE_SMALL=$(python3 -c \"\nimport json\nd = json.load(open('workspace/refinement/iter<N>/delta.json'))\nprint(d['consecutive_small'])\n\")\n```\n\nExit codes:\n- `0` — ACCEPT (overall improved or tied with non-negative net sub-axis, below the Accept band, no plateau)\n- `1` — REVERT (overall decreased)\n- `2` — REVERT (tied overall, but net sub-axis change negative)\n- `4` — HALT_PLATEAU (accepted but N consecutive iterations below threshold — stop early)\n- `5` — HALT_TARGET_MET (accepted AND reached the Accept band, overall ≥ 80 — stop)\n\nBehavior:\n\n- **ACCEPT (exit 0)**: keep `iter<N>/paper.tex` as the new best. Continue to iter N+1.\n- **REVERT (exit 1 or 2)**: copy `iter<N-1>/paper.tex` back as canonical, halt.\n- **HALT_PLATEAU (exit 4)**: keep current (it was accepted), but stop — further\n  iterations are unlikely to yield meaningful gains. In practice ~85% of\n  refinement gain comes in iteration 1; the plateau fires when subsequent\n  iterations improve by less than 1 point for 3 consecutive rounds.\n- **HALT_TARGET_MET (exit 5)**: keep current (it was accepted), but stop — the\n  paper has reached the Accept band (overall ≥ 80), so there is no reason to\n  keep iterating and risk a regression. The `delta.json` carries\n  `decision_band_prev` / `decision_band_curr` for the run report.\n\n**Override — DA CRITICAL.** If `concession_guard.py` (Step 1) returned exit 1\nfor this iteration, treat the outcome as **REVERT** even when `score_delta.py`\nsays ACCEPT: roll back to `iter<N-1>/paper.tex` and require the next revision to\naddress the standing CRITICAL finding.\n\nAlways log the decision via `apply_worklog.py --decision ...`.\n\n### 6. Halt rules\n\nHalt the loop when ANY of these is true:\n\n1. Iteration count reaches `ITER_CAP` (default 3).\n2. `score_delta.py` returned exit code 1 or 2 (REVERT), OR `concession_guard.py`\n   returned exit 1 (standing DA CRITICAL → forced REVERT).\n3. The simulated reviewer's `weaknesses` list is empty (no actionable\n   feedback to apply).\n4. `score_delta.py` returned exit code 4 (HALT_PLATEAU — plateau early-stop).\n5. `score_delta.py` returned exit code 5 (HALT_TARGET_MET — reached the Accept\n   band, overall ≥ 80; promote the current draft).\n\n### 7. Promote the best snapshot\n\nIdentify the iteration with the highest accepted `overall_score` (this may\nbe the latest accepted iteration, OR an earlier one if a later iteration\nwas reverted). Copy:\n\n```bash\ncp workspace/refinement/iter<best>/paper.tex workspace/final/paper.tex\ncp workspace/refinement/iter<best>/paper.pdf workspace/final/paper.pdf\n```\n\nThen in the final report, tell the user:\n- How many iterations were run\n- The final overall score and its decision band (Accept / Minor / Major / Reject)\n- The score trajectory with bands (e.g., \"iter0 58.0 Major → iter1 67.3 Minor (accept) → iter2 81.0 Accept (halt: target met)\")\n- Which iteration was promoted, and the halt reason (revert / plateau / target met / iter cap / DA critical)\n\n## Critical safety constraints (App. F.1 page 50–51)\n\nThe paper explicitly notes that early versions of the Refinement Agent\n\"exploited the automated reviewer's scoring function by superficially\nlisting missing baselines as limitations to artificially inflate\nacceptance scores.\" The verbatim prompt forbids this. **You must honor it:**\n\n- **[IRON RULE] Halt on score regression.** If `score_delta.py` returns exit\n  code 1 or 2 (REVERT), immediately revert to the previous snapshot and halt.\n  No further revision attempts are permitted after a regression.\n- **[IRON RULE] No new experiments in revision.** Ignore reviewer requests for\n  new experiments, ablations, or baselines. The Refinement Agent's job is\n  presentation, not new science. If the reviewer asks for missing data, simply\n  skip those points — do NOT add fabricated experiments, do NOT add a \"future\n  work\" item promising them.\n- **[IRON RULE] All numeric claims must match experimental_log.md.** The agent\n  cannot introduce new numbers, only re-present existing ones. Any number in\n  the revised paper that does not appear in experimental_log.md is a\n  hallucination.\n- **Never explicitly state a limitation.** The phrase \"we acknowledge as a\n  limitation that...\" is forbidden. The model can address weaknesses\n  through clearer explanation, but must not game the evaluator by listing\n  them defensively.\n\nThese rules prevent reward hacking and keep the refinement loop honest.\n\n## Resources\n\n- `references/prompt.md` — verbatim Content Refinement Agent prompt from App. F.1\n- `references/reviewer-rubric.md` — AgentReview-style scoring rubric (6 axes)\n- `references/halt-rules.md` — accept/revert/halt logic in formal pseudocode\n- `references/safe-revision-rules.md` — anti-reward-hack constraints\n- `references/writing-quality-check.md` — 5-category anti-AI-prose checklist (pointer to shared)\n- `references/ai-failure-modes.md` — 7-mode integrity gate run before first iteration (pointer to shared)\n- `references/da-reviewer.md` — Devil's Advocate reviewer protocol and concession rules\n- `scripts/score_delta.py` — accept/revert/halt decision from two score JSONs; emits decision bands + target-met halt (exit 5)\n- `scripts/decision_band.py` — map an overall score to a canonical decision band (Accept/Minor/Major/Reject)\n- `scripts/concession_guard.py` — enforce the DA concession-threshold protocol; blocks accept on a standing CRITICAL\n- `scripts/score_trajectory.py` — per-dimension score history, regression and plateau detection\n- `scripts/apply_worklog.py` — append iteration entries to worklog.json\n- `scripts/snapshot.py` — copy paper.tex/paper.pdf into iter<N>/ for rollback\n- `scripts/update_critique_memory.py` — **NEW** build/update critique_memory.json from worklog + review (AutoSci-inspired reviewer memory)\n- `skills/shared/writing_quality_check.md` — full anti-AI-prose checklist (5 categories)\n- `skills/shared/ai_failure_modes.md` — full AI research failure modes gate (7 modes)\n- `skills/shared/handoff_schemas.md` — formal data contracts between all pipeline steps\n- `skills/shared/research_brief_template.md` — **NEW** research brief schema (read §1–§4 before first reviewer call)","author":"@Ar9av","ownerProfile":null,"authorContacts":null,"sourceUrl":"https://github.com/Ar9av/PaperOrchestra/tree/main/skills/content-refinement-agent","license":"MIT","category":"writing","lang":"en","tokens":4162,"stars":0,"calls30d":2,"claimed":false,"visibility":"public","origin":"crawler","version":"0.1.0","createdAt":"2026-08-22","updatedAt":"2026-08-22","files":[{"path":"references/ai-failure-modes.md","size":813,"sha256":"1760e71645bc42095f51bcfa5c2d3dfcabd0cbf59339dbed11ed6e54b6bbe863"},{"path":"references/da-reviewer.md","size":3984,"sha256":"d118d0ffc91e54f0a9045c1faa041d5c797ea2d5cce5200bf0d9daf3586df4ad"},{"path":"references/halt-rules.md","size":6293,"sha256":"28c8812f5b85e857c43b9a5c1f8eca9323ef678bcd28cd3ea1b2d34e8ebb901f"},{"path":"references/prompt.md","size":5788,"sha256":"14d2b4070c8cbd1a01d7601bfbd62af2ce9208eaa2fe2b8532d108b03d5d4746"},{"path":"references/reviewer-rubric.md","size":6940,"sha256":"44a288b01a95ad823fb9eeb82635f41ac6e26a209c3034ab75d87457f640dfcf"},{"path":"references/safe-revision-rules.md","size":4953,"sha256":"8b479f8be3d70374199b4d910c3867f579b3136023fe48736f76758a023656b9"},{"path":"references/writing-quality-check.md","size":720,"sha256":"f78e0b3892066ac392f9af84058cd614632644684bdb32a8aed61f2de1cca79a"},{"path":"scripts/apply_worklog.py","size":3141,"sha256":"d4109fc1e3e2ca6d5d2a7d2e7ad7a39849e084f1359974e0a3394fbede462c70"},{"path":"scripts/concession_guard.py","size":7667,"sha256":"0548561fccfdf394af6c5df88d69bd297501000314edde2c1b32334b74fceade"},{"path":"scripts/decision_band.py","size":3722,"sha256":"371fdb7e353e7008536a2786e291bf233dbd99424cc120e29cfdcf3010bb5e6c"},{"path":"scripts/score_delta.py","size":7391,"sha256":"2177f1bf6f803f2a444fce462b632a2e7a708a1fd03fccbdeeab8475ba063b2a"},{"path":"scripts/score_trajectory.py","size":8796,"sha256":"53825d928c4dc4fb519ddf310d250a230c3627b2773baac18d66fc41c7f7eedc"},{"path":"scripts/snapshot.py","size":1621,"sha256":"16063fe9d283e37bd80ef00836fcaca6b3b3827c83a7269a238e22a31e169d66"},{"path":"scripts/update_critique_memory.py","size":11333,"sha256":"94e585b584ef77091448e6837c5eb714c207890adfb59e0c3998f7ec0c002475"}],"requires":{"mcp":[],"tools":[]},"safety":{"flags":[],"scannedAt":"2026-08-22","hasScripts":true,"networkEndpoints":[]}}