{"id":"llama-cpp","name":"llama-cpp","summary":"NVIDIAハードウェアを使わず、CPU、Apple Silicon、コンシューマーGPU上でLLM推論を実行します。","body":"# llama.cpp\n\nPure C/C++ LLM inference with minimal dependencies, optimized for CPUs and non-NVIDIA hardware.\n\n## When to use llama.cpp\n\n**Use llama.cpp when:**\n- Running on CPU-only machines\n- Deploying on Apple Silicon (M1/M2/M3/M4)\n- Using AMD or Intel GPUs (no CUDA)\n- Edge deployment (Raspberry Pi, embedded systems)\n- Need simple deployment without Docker/Python\n\n**Use TensorRT-LLM instead when:**\n- Have NVIDIA GPUs (A100/H100)\n- Need maximum throughput (100K+ tok/s)\n- Running in datacenter with CUDA\n\n**Use vLLM instead when:**\n- Have NVIDIA GPUs\n- Need Python-first API\n- Want PagedAttention\n\n## Quick start\n\n### Installation\n\n```bash\n# macOS/Linux\nbrew install llama.cpp\n\n# Or build from source\ngit clone https://github.com/ggerganov/llama.cpp\ncd llama.cpp\nmake\n\n# With Metal (Apple Silicon)\nmake LLAMA_METAL=1\n\n# With CUDA (NVIDIA)\nmake LLAMA_CUDA=1\n\n# With ROCm (AMD)\nmake LLAMA_HIP=1\n```\n\n### Download model\n\n```bash\n# Download from HuggingFace (GGUF format)\nhuggingface-cli download \\\n    TheBloke/Llama-2-7B-Chat-GGUF \\\n    llama-2-7b-chat.Q4_K_M.gguf \\\n    --local-dir models/\n\n# Or convert from HuggingFace\npython convert_hf_to_gguf.py models/llama-2-7b-chat/\n```\n\n### Run inference\n\n```bash\n# Simple chat\n./llama-cli \\\n    -m models/llama-2-7b-chat.Q4_K_M.gguf \\\n    -p \"Explain quantum computing\" \\\n    -n 256  # Max tokens\n\n# Interactive chat\n./llama-cli \\\n    -m models/llama-2-7b-chat.Q4_K_M.gguf \\\n    --interactive\n```\n\n### Server mode\n\n```bash\n# Start OpenAI-compatible server\n./llama-server \\\n    -m models/llama-2-7b-chat.Q4_K_M.gguf \\\n    --host 0.0.0.0 \\\n    --port 8080 \\\n    -ngl 32  # Offload 32 layers to GPU\n\n# Client request\ncurl http://localhost:8080/v1/chat/completions \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"model\": \"llama-2-7b-chat\",\n    \"messages\": [{\"role\": \"user\", \"content\": \"Hello!\"}],\n    \"temperature\": 0.7,\n    \"max_tokens\": 100\n  }'\n```\n\n## Quantization formats\n\n### GGUF format overview\n\n| Format | Bits | Size (7B) | Speed | Quality | Use Case |\n|--------|------|-----------|-------|---------|----------|\n| **Q4_K_M** | 4.5 | 4.1 GB | Fast | Good | **Recommended default** |\n| Q4_K_S | 4.3 | 3.9 GB | Faster | Lower | Speed critical |\n| Q5_K_M | 5.5 | 4.8 GB | Medium | Better | Quality critical |\n| Q6_K | 6.5 | 5.5 GB | Slower | Best | Maximum quality |\n| Q8_0 | 8.0 | 7.0 GB | Slow | Excellent | Minimal degradation |\n| Q2_K | 2.5 | 2.7 GB | Fastest | Poor | Testing only |\n\n### Choosing quantization\n\n```bash\n# General use (balanced)\nQ4_K_M  # 4-bit, medium quality\n\n# Maximum speed (more degradation)\nQ2_K or Q3_K_M\n\n# Maximum quality (slower)\nQ6_K or Q8_0\n\n# Very large models (70B, 405B)\nQ3_K_M or Q4_K_S  # Lower bits to fit in memory\n```\n\n## Hardware acceleration\n\n### Apple Silicon (Metal)\n\n```bash\n# Build with Metal\nmake LLAMA_METAL=1\n\n# Run with GPU acceleration (automatic)\n./llama-cli -m model.gguf -ngl 999  # Offload all layers\n\n# Performance: M3 Max 40-60 tokens/sec (Llama 2-7B Q4_K_M)\n```\n\n### NVIDIA GPUs (CUDA)\n\n```bash\n# Build with CUDA\nmake LLAMA_CUDA=1\n\n# Offload layers to GPU\n./llama-cli -m model.gguf -ngl 35  # Offload 35/40 layers\n\n# Hybrid CPU+GPU for large models\n./llama-cli -m llama-70b.Q4_K_M.gguf -ngl 20  # GPU: 20 layers, CPU: rest\n```\n\n### AMD GPUs (ROCm)\n\n```bash\n# Build with ROCm\nmake LLAMA_HIP=1\n\n# Run with AMD GPU\n./llama-cli -m model.gguf -ngl 999\n```\n\n## Common patterns\n\n### Batch processing\n\n```bash\n# Process multiple prompts from file\ncat prompts.txt | ./llama-cli \\\n    -m model.gguf \\\n    --batch-size 512 \\\n    -n 100\n```\n\n### Constrained generation\n\n```bash\n# JSON output with grammar\n./llama-cli \\\n    -m model.gguf \\\n    -p \"Generate a person: \" \\\n    --grammar-file grammars/json.gbnf\n\n# Outputs valid JSON only\n```\n\n### Context size\n\n```bash\n# Increase context (default 512)\n./llama-cli \\\n    -m model.gguf \\\n    -c 4096  # 4K context window\n\n# Very long context (if model supports)\n./llama-cli -m model.gguf -c 32768  # 32K context\n```\n\n## Performance benchmarks\n\n### CPU performance (Llama 2-7B Q4_K_M)\n\n| CPU | Threads | Speed | Cost |\n|-----|---------|-------|------|\n| Apple M3 Max | 16 | 50 tok/s | $0 (local) |\n| AMD Ryzen 9 7950X | 32 | 35 tok/s | $0.50/hour |\n| Intel i9-13900K | 32 | 30 tok/s | $0.40/hour |\n| AWS c7i.16xlarge | 64 | 40 tok/s | $2.88/hour |\n\n### GPU acceleration (Llama 2-7B Q4_K_M)\n\n| GPU | Speed | vs CPU | Cost |\n|-----|-------|--------|------|\n| NVIDIA RTX 4090 | 120 tok/s | 3-4× | $0 (local) |\n| NVIDIA A10 | 80 tok/s | 2-3× | $1.00/hour |\n| AMD MI250 | 70 tok/s | 2× | $2.00/hour |\n| Apple M3 Max (Metal) | 50 tok/s | ~Same | $0 (local) |\n\n## Supported models\n\n**LLaMA family**:\n- Llama 2 (7B, 13B, 70B)\n- Llama 3 (8B, 70B, 405B)\n- Code Llama\n\n**Mistral family**:\n- Mistral 7B\n- Mixtral 8x7B, 8x22B\n\n**Other**:\n- Falcon, BLOOM, GPT-J\n- Phi-3, Gemma, Qwen\n- LLaVA (vision), Whisper (audio)\n\n**Find models**: https://huggingface.co/models?library=gguf\n\n## References\n\n- **[Quantization Guide](references/quantization.md)** - GGUF formats, conversion, quality comparison\n- **[Server Deployment](references/server.md)** - API endpoints, Docker, monitoring\n- **[Optimization](references/optimization.md)** - Performance tuning, hybrid CPU+GPU\n\n## Resources\n\n- **GitHub**: https://github.com/ggerganov/llama.cpp\n- **Models**: https://huggingface.co/models?library=gguf\n- **Discord**: https://discord.gg/llama-cpp","author":"@Orchestra-Research","ownerProfile":null,"authorContacts":null,"sourceUrl":"https://github.com/Orchestra-Research/AI-Research-SKILLs/tree/main/12-inference-serving/llama-cpp","license":"MIT","category":"coding","lang":"en","tokens":1781,"stars":0,"calls30d":2,"claimed":false,"visibility":"public","origin":"crawler","version":"0.1.0","createdAt":"2026-08-22","updatedAt":"2026-08-22","files":[{"path":"references/optimization.md","size":1659,"sha256":"4e89811c0827f5eb9deb9fc609cd441eb197fb0fa85e79dd8c1f272b14a669fd"},{"path":"references/quantization.md","size":4956,"sha256":"c7686a86893d4e0e052cc733b62ea1bc0e5dae8f1b174436cc293a46a486893d"},{"path":"references/server.md","size":2259,"sha256":"181a84ae177f305205cc30d05305fc4237aa255f3fff29299362cb1e0cb6db64"}],"requires":{"mcp":[],"tools":[]},"safety":{"flags":[],"scannedAt":"2026-08-22","hasScripts":false,"networkEndpoints":["discord.gg","huggingface.co"]}}