{"id":"pytorch-lightning","name":"pytorch-lightning","summary":"Trainerクラス、自動分散トレーニング(DDP/FSDP/DeepSpeed)、コールバックシステム、最小限のボイラープレートを備えた高レベルPyTorchフレームワーク。","body":"# PyTorch Lightning - High-Level Training Framework\n\n## Quick start\n\nPyTorch Lightning organizes PyTorch code to eliminate boilerplate while maintaining flexibility.\n\n**Installation**:\n```bash\npip install lightning\n```\n\n**Convert PyTorch to Lightning** (3 steps):\n\n```python\nimport lightning as L\nimport torch\nfrom torch import nn\nfrom torch.utils.data import DataLoader, Dataset\n\n# Step 1: Define LightningModule (organize your PyTorch code)\nclass LitModel(L.LightningModule):\n    def __init__(self, hidden_size=128):\n        super().__init__()\n        self.model = nn.Sequential(\n            nn.Linear(28 * 28, hidden_size),\n            nn.ReLU(),\n            nn.Linear(hidden_size, 10)\n        )\n\n    def training_step(self, batch, batch_idx):\n        x, y = batch\n        y_hat = self.model(x)\n        loss = nn.functional.cross_entropy(y_hat, y)\n        self.log('train_loss', loss)  # Auto-logged to TensorBoard\n        return loss\n\n    def configure_optimizers(self):\n        return torch.optim.Adam(self.parameters(), lr=1e-3)\n\n# Step 2: Create data\ntrain_loader = DataLoader(train_dataset, batch_size=32)\n\n# Step 3: Train with Trainer (handles everything else!)\ntrainer = L.Trainer(max_epochs=10, accelerator='gpu', devices=2)\nmodel = LitModel()\ntrainer.fit(model, train_loader)\n```\n\n**That's it!** Trainer handles:\n- GPU/TPU/CPU switching\n- Distributed training (DDP, FSDP, DeepSpeed)\n- Mixed precision (FP16, BF16)\n- Gradient accumulation\n- Checkpointing\n- Logging\n- Progress bars\n\n## Common workflows\n\n### Workflow 1: From PyTorch to Lightning\n\n**Original PyTorch code**:\n```python\nmodel = MyModel()\noptimizer = torch.optim.Adam(model.parameters())\nmodel.to('cuda')\n\nfor epoch in range(max_epochs):\n    for batch in train_loader:\n        batch = batch.to('cuda')\n        optimizer.zero_grad()\n        loss = model(batch)\n        loss.backward()\n        optimizer.step()\n```\n\n**Lightning version**:\n```python\nclass LitModel(L.LightningModule):\n    def __init__(self):\n        super().__init__()\n        self.model = MyModel()\n\n    def training_step(self, batch, batch_idx):\n        loss = self.model(batch)  # No .to('cuda') needed!\n        return loss\n\n    def configure_optimizers(self):\n        return torch.optim.Adam(self.parameters())\n\n# Train\ntrainer = L.Trainer(max_epochs=10, accelerator='gpu')\ntrainer.fit(LitModel(), train_loader)\n```\n\n**Benefits**: 40+ lines → 15 lines, no device management, automatic distributed\n\n### Workflow 2: Validation and testing\n\n```python\nclass LitModel(L.LightningModule):\n    def __init__(self):\n        super().__init__()\n        self.model = MyModel()\n\n    def training_step(self, batch, batch_idx):\n        x, y = batch\n        y_hat = self.model(x)\n        loss = nn.functional.cross_entropy(y_hat, y)\n        self.log('train_loss', loss)\n        return loss\n\n    def validation_step(self, batch, batch_idx):\n        x, y = batch\n        y_hat = self.model(x)\n        val_loss = nn.functional.cross_entropy(y_hat, y)\n        acc = (y_hat.argmax(dim=1) == y).float().mean()\n        self.log('val_loss', val_loss)\n        self.log('val_acc', acc)\n\n    def test_step(self, batch, batch_idx):\n        x, y = batch\n        y_hat = self.model(x)\n        test_loss = nn.functional.cross_entropy(y_hat, y)\n        self.log('test_loss', test_loss)\n\n    def configure_optimizers(self):\n        return torch.optim.Adam(self.parameters(), lr=1e-3)\n\n# Train with validation\ntrainer = L.Trainer(max_epochs=10)\ntrainer.fit(model, train_loader, val_loader)\n\n# Test\ntrainer.test(model, test_loader)\n```\n\n**Automatic features**:\n- Validation runs every epoch by default\n- Metrics logged to TensorBoard\n- Best model checkpointing based on val_loss\n\n### Workflow 3: Distributed training (DDP)\n\n```python\n# Same code as single GPU!\nmodel = LitModel()\n\n# 8 GPUs with DDP (automatic!)\ntrainer = L.Trainer(\n    accelerator='gpu',\n    devices=8,\n    strategy='ddp'  # Or 'fsdp', 'deepspeed'\n)\n\ntrainer.fit(model, train_loader)\n```\n\n**Launch**:\n```bash\n# Single command, Lightning handles the rest\npython train.py\n```\n\n**No changes needed**:\n- Automatic data distribution\n- Gradient synchronization\n- Multi-node support (just set `num_nodes=2`)\n\n### Workflow 4: Callbacks for monitoring\n\n```python\nfrom lightning.pytorch.callbacks import ModelCheckpoint, EarlyStopping, LearningRateMonitor\n\n# Create callbacks\ncheckpoint = ModelCheckpoint(\n    monitor='val_loss',\n    mode='min',\n    save_top_k=3,\n    filename='model-{epoch:02d}-{val_loss:.2f}'\n)\n\nearly_stop = EarlyStopping(\n    monitor='val_loss',\n    patience=5,\n    mode='min'\n)\n\nlr_monitor = LearningRateMonitor(logging_interval='epoch')\n\n# Add to Trainer\ntrainer = L.Trainer(\n    max_epochs=100,\n    callbacks=[checkpoint, early_stop, lr_monitor]\n)\n\ntrainer.fit(model, train_loader, val_loader)\n```\n\n**Result**:\n- Auto-saves best 3 models\n- Stops early if no improvement for 5 epochs\n- Logs learning rate to TensorBoard\n\n### Workflow 5: Learning rate scheduling\n\n```python\nclass LitModel(L.LightningModule):\n    # ... (training_step, etc.)\n\n    def configure_optimizers(self):\n        optimizer = torch.optim.Adam(self.parameters(), lr=1e-3)\n\n        # Cosine annealing\n        scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(\n            optimizer,\n            T_max=100,\n            eta_min=1e-5\n        )\n\n        return {\n            'optimizer': optimizer,\n            'lr_scheduler': {\n                'scheduler': scheduler,\n                'interval': 'epoch',  # Update per epoch\n                'frequency': 1\n            }\n        }\n\n# Learning rate auto-logged!\ntrainer = L.Trainer(max_epochs=100)\ntrainer.fit(model, train_loader)\n```\n\n## When to use vs alternatives\n\n**Use PyTorch Lightning when**:\n- Want clean, organized code\n- Need production-ready training loops\n- Switching between single GPU, multi-GPU, TPU\n- Want built-in callbacks and logging\n- Team collaboration (standardized structure)\n\n**Key advantages**:\n- **Organized**: Separates research code from engineering\n- **Automatic**: DDP, FSDP, DeepSpeed with 1 line\n- **Callbacks**: Modular training extensions\n- **Reproducible**: Less boilerplate = fewer bugs\n- **Tested**: 1M+ downloads/month, battle-tested\n\n**Use alternatives instead**:\n- **Accelerate**: Minimal changes to existing code, more flexibility\n- **Ray Train**: Multi-node orchestration, hyperparameter tuning\n- **Raw PyTorch**: Maximum control, learning purposes\n- **Keras**: TensorFlow ecosystem\n\n## Common issues\n\n**Issue: Loss not decreasing**\n\nCheck data and model setup:\n```python\n# Add to training_step\ndef training_step(self, batch, batch_idx):\n    if batch_idx == 0:\n        print(f\"Batch shape: {batch[0].shape}\")\n        print(f\"Labels: {batch[1]}\")\n    loss = ...\n    return loss\n```\n\n**Issue: Out of memory**\n\nReduce batch size or use gradient accumulation:\n```python\ntrainer = L.Trainer(\n    accumulate_grad_batches=4,  # Effective batch = batch_size × 4\n    precision='bf16'  # Or 'fp16', reduces memory 50%\n)\n```\n\n**Issue: Validation not running**\n\nEnsure you pass val_loader:\n```python\n# WRONG\ntrainer.fit(model, train_loader)\n\n# CORRECT\ntrainer.fit(model, train_loader, val_loader)\n```\n\n**Issue: DDP spawns multiple processes unexpectedly**\n\nLightning auto-detects GPUs. Explicitly set devices:\n```python\n# Test on CPU first\ntrainer = L.Trainer(accelerator='cpu', devices=1)\n\n# Then GPU\ntrainer = L.Trainer(accelerator='gpu', devices=1)\n```\n\n## Advanced topics\n\n**Callbacks**: See [references/callbacks.md](references/callbacks.md) for EarlyStopping, ModelCheckpoint, custom callbacks, and callback hooks.\n\n**Distributed strategies**: See [references/distributed.md](references/distributed.md) for DDP, FSDP, DeepSpeed ZeRO integration, multi-node setup.\n\n**Hyperparameter tuning**: See [references/hyperparameter-tuning.md](references/hyperparameter-tuning.md) for integration with Optuna, Ray Tune, and WandB sweeps.\n\n## Hardware requirements\n\n- **CPU**: Works (good for debugging)\n- **Single GPU**: Works\n- **Multi-GPU**: DDP (default), FSDP, or DeepSpeed\n- **Multi-node**: DDP, FSDP, DeepSpeed\n- **TPU**: Supported (8 cores)\n- **Apple MPS**: Supported\n\n**Precision options**:\n- FP32 (default)\n- FP16 (V100, older GPUs)\n- BF16 (A100/H100, recommended)\n- FP8 (H100)\n\n## Resources\n\n- Docs: https://lightning.ai/docs/pytorch/stable/\n- GitHub: https://github.com/Lightning-AI/pytorch-lightning ⭐ 29,000+\n- Version: 2.5.5+\n- Examples: https://github.com/Lightning-AI/pytorch-lightning/tree/master/examples\n- Discord: https://discord.gg/lightning-ai\n- Used by: Kaggle winners, research labs, production teams","author":"@Orchestra-Research","ownerProfile":null,"authorContacts":null,"sourceUrl":"https://github.com/Orchestra-Research/AI-Research-SKILLs/tree/main/08-distributed-training/pytorch-lightning","license":"MIT","category":"coding","lang":"en","tokens":2125,"stars":0,"calls30d":2,"claimed":false,"visibility":"public","origin":"crawler","version":"0.1.0","createdAt":"2026-08-22","updatedAt":"2026-08-22","files":[{"path":"references/callbacks.md","size":12031,"sha256":"2940b978bdec21b6f8da54c72d1858015e5ba290f9b05f1a659d67c25860e259"},{"path":"references/distributed.md","size":10818,"sha256":"c7d183b74a48550541658ace014c6ef8fa15f2c625075369e4ad644ad950ecdf"},{"path":"references/hyperparameter-tuning.md","size":12517,"sha256":"fb733b9a324d4bb0c941c70b9bb01c59785d53a1827b48db5274520e1eb66e16"}],"requires":{"mcp":[],"tools":[]},"safety":{"flags":[{"code":"code.eval","kind":"dangerous-code","where":"references/distributed.md:353","excerpt":"eval(","message":"evaluates code at runtime","severity":"warn"}],"scannedAt":"2026-08-22","hasScripts":false,"networkEndpoints":["discord.gg","docs.ray.io","docs.wandb.ai","lightning.ai","optuna.readthedocs.io"]}}