{"id":"skill-monitor","name":"skill-monitor","summary":"セッション全体でのスキル効果を分析しましょう。スキルごとの指標(行動率、摩擦、成果)を計算し、劣化するスキルを特定し、改善提案を生成します。","body":"# Skill Monitor\n\nClosed-loop skill effectiveness monitoring. Reads session metrics,\ncomputes per-skill signals, identifies what's working and what needs\nimprovement.\n\nInspired by the deploy-monitor-evaluate-improve feedback loop:\nskills get better over time instead of staying static.\n\n## Requirements\n\nRequires `.claude/session-metrics/metrics.jsonl` from `/session-scan`.\nIf no data: suggest running `/session-scan` first.\n\n## Usage\n\n```\n/skill-monitor                     # Dashboard: all skills\n/skill-monitor --skill review      # Deep-dive on one skill\n/skill-monitor --improve           # Generate improvement recommendations\n/skill-monitor --window 30d        # Change comparison window (default: 7d)\n```\n\n## What Main Context Does\n\n### Step 1: Parse Arguments\n\nExtract from `$ARGUMENTS`:\n\n- **`--skill NAME`**: Focus on one skill (e.g., `review`, `plan`, `investigate`)\n- **`--improve`**: Spawn analysis agent for improvement recommendations\n- **`--window PERIOD`**: Comparison window (`7d`, `30d`, `all`; default: `7d`)\n\n### Step 2: Load Metrics\n\nRead `.claude/session-metrics/metrics.jsonl`. For each entry, extract\nthe `skill_effectiveness` field (added by compute-metrics.py v2).\n\nFilter by window period. Count sessions with and without skill usage.\n\nIf no `skill_effectiveness` data exists in metrics: \"Metrics were\ncomputed before skill tracking was added. Run `/session-scan --rescan`\nto recompute.\"\n\n**OTel `invocation_trigger` (CC v2.1.126+)**: when `compute-metrics.py`\ningests `claude_code.skill_activated` events, each invocation carries\nan `invocation_trigger` of `\"user-slash\"`, `\"claude-proactive\"`, or\n`\"nested-skill\"`. If absent (older sessions), default to\n`\"unknown\"` — do NOT assume `\"user-slash\"`.\n\n### Step 3: Compute Per-Skill Aggregates\n\nFor each skill found across all sessions, aggregate:\n\n```\n| Metric                  | Computation                                    |\n|-------------------------|------------------------------------------------|\n| Total invocations       | Sum of invocation_count across sessions        |\n| Sessions used in        | Count of sessions containing this skill        |\n| Action rate             | Weighted avg of per-session action_rate         |\n| Avg post-errors         | Weighted avg of avg_post_errors                |\n| Avg post-corrections    | Weighted avg of avg_post_corrections           |\n| Outcome distribution    | Count of effective/friction/no_action/mixed    |\n| Effectiveness score     | action_rate - (0.3 * avg_post_corrections)     |\n| Adjusted score          | For analysis/check skills, use lower thresholds |\n| Trigger distribution    | Counts of user-slash / claude-proactive / nested-skill / unknown |\n| Proactive trigger rate  | claude-proactive / (user-slash + claude-proactive + nested-skill) |\n| Auto-load gap           | Skills with 0 claude-proactive invocations across window |\n```\n\n**Auto-load gap detection (CC v2.1.126+)**: Skills with `auto-loaded`\nbehavior in their description (i.e., not `disable-model-invocation: true`)\nare EXPECTED to fire as `claude-proactive`. A skill that is ONLY ever\ninvoked via `user-slash` is failing its description's routing intent.\nFlag any auto-loadable skill where `proactive_trigger_rate == 0` over\nthe window. This is the structural answer to the \"zero skill\nauto-loading\" gap from the 137-session analysis (see MEMORY.md).\n**Confidence floor**: only flag if total invocations >= 5 in window.\n\n**Skill type weighting**: Analysis and check skills (verify, triage,\nperf, boundaries, pr-review, audit) have low action rates BY DESIGN —\ntheir success is \"found issues\" or \"confirmed things pass\". Apply\nadjusted thresholds:\n\n| Skill Type | Flag Threshold | Expected Action Rate |\n|------------|---------------|---------------------|\n| Execution (work, quick, full) | < 0.5 | > 0.7 |\n| Analysis (perf, boundaries, audit, pr-review) | < 0.3 | 0.3-0.5 |\n| Check (verify, triage) | < 0.1 | 0.0-0.3 |\n| Knowledge (compound, learn, brief) | < 0.5 | > 0.5 |\n\nAlso compute **baseline friction** (avg friction of sessions WITHOUT\nany skill usage) vs **skill friction** (avg friction of sessions\nWITH skill usage). Delta = skill_friction - baseline_friction.\nNegative delta = skills reduce friction (good).\n\n### Step 4: Display Dashboard\n\n**Dashboard mode** (no `--skill`):\n\n```\n## Skill Effectiveness Dashboard (last {window})\n\nBaseline friction (no skills): 0.32 | With skills: 0.18 | Delta: -0.14\n\n| Skill           | Uses | Sessions | Slash/Proactive/Nested | Action% | Errors | Corr | Outcome   | Score |\n|-----------------|------|----------|------------------------|---------|--------|------|-----------|-------|\n| /phx:review     | 12   | 8        |    8 /  3 /  1         | 92%     | 0.5    | 0.1  | effective | 0.89  |\n| /phx:plan       | 9    | 7        |    9 /  0 /  0         | 100%    | 0.2    | 0.0  | effective | 1.00  |\n| /phx:investigate| 5    | 5        |    5 /  0 /  0         | 80%     | 1.2    | 0.4  | mixed     | 0.68  |\n\nSkills needing attention:\n- /phx:investigate (high post-errors)\n- /phx:plan (auto-load gap — 0/9 proactive; description not routing)\n```\n\nFlag skills using type-adjusted thresholds (see weighting table above).\nAlso flag if avg_post_corrections > 1 or outcome is predominantly \"friction\".\n**Also flag auto-load gap**: auto-loadable skills (without\n`disable-model-invocation: true`) with proactive_trigger_rate == 0 and\ntotal invocations >= 5. This is a description/routing problem — the skill\nexists but Claude isn't loading it on its own.\n\nWhen displaying flagged skills, note if the flag is \"expected\" for the\nskill type (e.g., verify at 0.24 is normal for a check skill).\n\n**Skill deep-dive** (`--skill NAME`):\n\nShow per-session breakdown for that skill, including session IDs,\ndates, individual outcome signals, AND `invocation_trigger` per\ninvocation. If a skill is dominated by `user-slash` triggers, surface\nwhich 1-3 description keywords might unlock proactive routing —\ncross-reference against the skill's current description in\n`plugins/elixir-phoenix/skills/{name}/SKILL.md`. If session reports\nexist in `.claude/session-analysis/`, reference them.\n\n### Step 5: Improvement Mode (--improve)\n\nSpawn `skill-effectiveness-analyzer` agent:\n\n```\nAgent(subagent_type=\"skill-effectiveness-analyzer\", model=\"sonnet\", prompt=\"\"\"\nAnalyze skill effectiveness data and recommend improvements.\n\nMetrics data: {aggregated_metrics_json}\n\nSessions with friction outcomes: {session_ids}\n\nFor each underperforming skill:\n1. Identify failure patterns from outcome signals\n2. Propose specific skill/agent changes\n3. Suggest new Iron Laws if patterns are systematic\n\nWrite recommendations to: .claude/skill-metrics/recommendations-{date}.md\n\"\"\")\n```\n\n### Step 6: Write Output\n\nWrite aggregated metrics to `.claude/skill-metrics/dashboard-{date}.json`:\n\n```json\n{\n  \"computed_at\": \"2026-03-03T14:00:00Z\",\n  \"window\": \"7d\",\n  \"baseline_friction\": 0.32,\n  \"skill_friction\": 0.18,\n  \"friction_delta\": -0.14,\n  \"skills\": {\n    \"/phx:plan\": {\n      \"invocations\": 9,\n      \"trigger_distribution\": {\n        \"user-slash\": 9,\n        \"claude-proactive\": 0,\n        \"nested-skill\": 0,\n        \"unknown\": 0\n      },\n      \"proactive_trigger_rate\": 0.0,\n      \"auto_load_gap\": true\n    }\n  },\n  \"flagged_skills\": [\"investigate\", \"plan:auto-load-gap\"]\n}\n```\n\nAppend-only: never modify previous dashboard files.\n\n## Iron Laws\n\n1. **NEVER modify metrics.jsonl** — read-only from this skill\n2. **Baseline comparison is mandatory** — raw numbers without baseline are meaningless\n3. **Flag, don't judge** — surface data, let the human decide what to fix\n4. **Evidence tags on recommendations** — every suggestion needs session citations\n5. **Trigger source must not be inferred** — only treat invocations as\n   `user-slash` / `claude-proactive` / `nested-skill` when the OTel\n   `invocation_trigger` attribute is present (CC v2.1.126+). Older\n   sessions use `\"unknown\"`; never silently bucket them as user-slash —\n   it would hide the auto-load gap.\n\n## Integration\n\n```\n/session-scan → metrics.jsonl (with skill_effectiveness)\n       ↓\n/skill-monitor → dashboard + flagged skills\n       ↓\n/skill-monitor --improve → recommendations\n       ↓\nDeveloper updates skills/agents → deploy → repeat\n```\n\n## References\n\n- `references/effectiveness-metrics.md` — Full metrics schema and evaluation criteria\n- `references/improvement-template.md` — Template for improvement recommendations","author":"@oliver-kriska","ownerProfile":null,"authorContacts":null,"sourceUrl":"https://github.com/oliver-kriska/claude-elixir-phoenix/tree/main/.claude/skills/skill-monitor","license":"MIT","category":"review","lang":"en","tokens":2155,"stars":0,"calls30d":2,"claimed":false,"visibility":"public","origin":"crawler","version":"0.1.0","createdAt":"2026-08-22","updatedAt":"2026-08-22","files":[{"path":"references/effectiveness-metrics.md","size":9018,"sha256":"2b82d6fee52be05331928cbe3b8d8fc9ae1454494d7588606d5b4ff47565f55a"},{"path":"references/improvement-template.md","size":4337,"sha256":"849c1a02534d827e9a2d6ae2063bb83ca55847a20f855178c4b920402038877d"}],"requires":{"mcp":[],"tools":[]},"safety":{"flags":[],"scannedAt":"2026-08-22","hasScripts":false,"networkEndpoints":[]}}