{"id":"rwkv","name":"rwkv-architecture","summary":"RNN+トランスハイブリッドでO(n)推論を行った。線形時間、無限のコンテキスト、KVキャッシュなし。","body":"# RWKV - Receptance Weighted Key Value\n\n## Quick start\n\nRWKV (RwaKuv) combines Transformer parallelization (training) with RNN efficiency (inference).\n\n**Installation**:\n```bash\n# Install PyTorch\npip install torch --upgrade --extra-index-url https://download.pytorch.org/whl/cu121\n\n# Install dependencies\npip install pytorch-lightning==1.9.5 deepspeed wandb ninja --upgrade\n\n# Install RWKV\npip install rwkv\n```\n\n**Basic usage** (GPT mode + RNN mode):\n```python\nimport os\nfrom rwkv.model import RWKV\n\nos.environ[\"RWKV_JIT_ON\"] = '1'\nos.environ[\"RWKV_CUDA_ON\"] = '1'  # Use CUDA kernel for speed\n\n# Load model\nmodel = RWKV(\n    model='/path/to/RWKV-4-Pile-1B5-20220903-8040',\n    strategy='cuda fp16'\n)\n\n# GPT mode (parallel processing)\nout, state = model.forward([187, 510, 1563, 310, 247], None)\nprint(out.detach().cpu().numpy())  # Logits\n\n# RNN mode (sequential processing, same result)\nout, state = model.forward([187, 510], None)  # First 2 tokens\nout, state = model.forward([1563], state)      # Next token\nout, state = model.forward([310, 247], state)  # Last tokens\nprint(out.detach().cpu().numpy())  # Same logits as above!\n```\n\n## Common workflows\n\n### Workflow 1: Text generation (streaming)\n\n**Efficient token-by-token generation**:\n```python\nfrom rwkv.model import RWKV\nfrom rwkv.utils import PIPELINE\n\nmodel = RWKV(model='RWKV-4-Pile-14B-20230313-ctx8192-test1050', strategy='cuda fp16')\npipeline = PIPELINE(model, \"20B_tokenizer.json\")\n\n# Initial prompt\nprompt = \"The future of AI is\"\nstate = None\n\n# Generate token by token\nfor token in prompt:\n    out, state = pipeline.model.forward(pipeline.encode(token), state)\n\n# Continue generation\nfor _ in range(100):\n    out, state = pipeline.model.forward(None, state)\n    token = pipeline.sample_logits(out)\n    print(pipeline.decode(token), end='', flush=True)\n```\n\n**Key advantage**: Constant memory per token (no growing KV cache)\n\n### Workflow 2: Long context processing (infinite context)\n\n**Process million-token sequences**:\n```python\nmodel = RWKV(model='RWKV-4-Pile-14B', strategy='cuda fp16')\n\n# Process very long document\nstate = None\nlong_document = load_document()  # e.g., 1M tokens\n\n# Stream through entire document\nfor chunk in chunks(long_document, chunk_size=1024):\n    out, state = model.forward(chunk, state)\n\n# State now contains information from entire 1M token document\n# Memory usage: O(1) (constant, not O(n)!)\n```\n\n### Workflow 3: Fine-tuning RWKV\n\n**Standard fine-tuning workflow**:\n```python\n# Training script\nimport pytorch_lightning as pl\nfrom rwkv.model import RWKV\nfrom rwkv.trainer import RWKVTrainer\n\n# Configure model\nconfig = {\n    'n_layer': 24,\n    'n_embd': 1024,\n    'vocab_size': 50277,\n    'ctx_len': 1024\n}\n\n# Setup trainer\ntrainer = pl.Trainer(\n    accelerator='gpu',\n    devices=8,\n    precision='bf16',\n    strategy='deepspeed_stage_2',\n    max_epochs=1\n)\n\n# Train\nmodel = RWKV(config)\ntrainer.fit(model, train_dataloader)\n```\n\n### Workflow 4: RWKV vs Transformer comparison\n\n**Memory comparison** (1M token sequence):\n```python\n# Transformer (GPT)\n# Memory: O(n²) for attention\n# KV cache: 1M × hidden_dim × n_layers × 2 (keys + values)\n# Example: 1M × 4096 × 24 × 2 = ~400GB (impractical!)\n\n# RWKV\n# Memory: O(1) per token\n# State: hidden_dim × n_layers = 4096 × 24 = ~400KB\n# 1,000,000× more efficient!\n```\n\n**Speed comparison** (inference):\n```python\n# Transformer: O(n) per token (quadratic overall)\n# First token: 1 computation\n# Second token: 2 computations\n# ...\n# 1000th token: 1000 computations\n\n# RWKV: O(1) per token (linear overall)\n# Every token: 1 computation\n# 1000th token: 1 computation (same as first!)\n```\n\n## When to use vs alternatives\n\n**Use RWKV when**:\n- Need very long context (100K+ tokens)\n- Want constant memory usage\n- Building streaming applications\n- Need RNN efficiency with Transformer performance\n- Memory-constrained deployment\n\n**Key advantages**:\n- **Linear time**: O(n) vs O(n²) for Transformers\n- **No KV cache**: Constant memory per token\n- **Infinite context**: No fixed window limit\n- **Parallelizable training**: Like GPT\n- **Sequential inference**: Like RNN\n\n**Use alternatives instead**:\n- **Transformers**: Need absolute best performance, have compute\n- **Mamba**: Want state-space models\n- **RetNet**: Need retention mechanism\n- **Hyena**: Want convolution-based approach\n\n## Common issues\n\n**Issue: Out of memory during training**\n\nUse gradient checkpointing and DeepSpeed:\n```python\ntrainer = pl.Trainer(\n    strategy='deepspeed_stage_3',  # Full ZeRO-3\n    precision='bf16'\n)\n```\n\n**Issue: Slow inference**\n\nEnable CUDA kernel:\n```python\nos.environ[\"RWKV_CUDA_ON\"] = '1'\n```\n\n**Issue: Model not loading**\n\nCheck model path and strategy:\n```python\nmodel = RWKV(\n    model='/absolute/path/to/model.pth',\n    strategy='cuda fp16'  # Or 'cpu fp32' for CPU\n)\n```\n\n**Issue: State management in RNN mode**\n\nAlways pass state between forward calls:\n```python\n# WRONG: State lost\nout1, _ = model.forward(tokens1, None)\nout2, _ = model.forward(tokens2, None)  # No context from tokens1!\n\n# CORRECT: State preserved\nout1, state = model.forward(tokens1, None)\nout2, state = model.forward(tokens2, state)  # Has context from tokens1\n```\n\n## Advanced topics\n\n**Time-mixing and channel-mixing**: See [references/architecture-details.md](references/architecture-details.md) for WKV operation, time-decay mechanism, and receptance gates.\n\n**State management**: See [references/state-management.md](references/state-management.md) for att_x_prev, att_kv, ffn_x_prev states, and numerical stability considerations.\n\n**RWKV-7 improvements**: See [references/rwkv7.md](references/rwkv7.md) for latest architectural improvements (March 2025) and multimodal capabilities.\n\n## Hardware requirements\n\n- **GPU**: NVIDIA (CUDA 11.6+) or CPU\n- **VRAM** (FP16):\n  - 169M model: 1GB\n  - 430M model: 2GB\n  - 1.5B model: 4GB\n  - 3B model: 8GB\n  - 7B model: 16GB\n  - 14B model: 32GB\n- **Inference**: O(1) memory per token\n- **Training**: Parallelizable like GPT\n\n**Performance** (vs Transformers):\n- **Speed**: Similar training, faster inference\n- **Memory**: 1000× less for long sequences\n- **Scaling**: Linear vs quadratic\n\n## Resources\n\n- Paper (RWKV): https://arxiv.org/abs/2305.13048 (May 2023)\n- Paper (RWKV-7): https://arxiv.org/abs/2503.14456 (March 2025)\n- GitHub: https://github.com/BlinkDL/RWKV-LM ⭐ 12,000+\n- Docs: https://wiki.rwkv.com/\n- Models: https://huggingface.co/BlinkDL\n- Linux Foundation AI: Official project\n- Production: Microsoft Windows, Office integration, NeMo support","author":"@Orchestra-Research","ownerProfile":null,"authorContacts":null,"sourceUrl":"https://github.com/Orchestra-Research/AI-Research-SKILLs/tree/main/01-model-architecture/rwkv","license":"MIT","category":"coding","lang":"en","tokens":1852,"stars":0,"calls30d":2,"claimed":false,"visibility":"public","origin":"crawler","version":"0.1.0","createdAt":"2026-08-22","updatedAt":"2026-08-22","files":[{"path":"references/architecture-details.md","size":9304,"sha256":"8d49b47e13d78771fb756b47eb6bca6c4aa9f85ffa5d4916eec5826356955487"},{"path":"references/rwkv7.md","size":10428,"sha256":"dffae18583757ebeefabdf58961577ecd9e8e43d601db942385ccb4e0295fa45"},{"path":"references/state-management.md","size":9497,"sha256":"c4de4fed3a84e4b515485e6a3e0742e0645ac02c191dd1cd2a4fc2a2676e4a6c"}],"requires":{"mcp":[],"tools":[]},"safety":{"flags":[],"scannedAt":"2026-08-22","hasScripts":false,"networkEndpoints":["arxiv.org","discord.gg","download.pytorch.org","huggingface.co","wiki.rwkv.com"]}}