{"id":"administering-linux","name":"administering-linux","summary":"Linuxシステムの管理は、systemdサービス、プロセス管理、ファイルシステム、ネットワーク、パフォーマンスチューニング、トラブルシューティングをカバーします。","body":"# Linux Administration\n\nComprehensive Linux system administration for managing servers, deploying applications, and troubleshooting production issues in modern cloud-native environments.\n\n## Purpose\n\nThis skill teaches fundamental and intermediate Linux administration for DevOps engineers, SREs, backend developers, and platform engineers. Focus on systemd-based distributions (Ubuntu, RHEL, Debian, Fedora) covering service management, process monitoring, filesystem operations, user administration, performance tuning, log analysis, and network configuration.\n\nModern infrastructure requires solid Linux fundamentals even with containerization. Container hosts run Linux, Kubernetes nodes need optimization, and troubleshooting production issues requires understanding systemd, processes, and logs.\n\n**Not Covered:**\n- Advanced networking (BGP, OSPF) - see `network-architecture` skill\n- Deep security hardening (compliance, pentesting) - see `security-hardening` skill\n- Configuration management at scale (Ansible, Puppet) - see `configuration-management` skill\n- Container orchestration - see `kubernetes-operations` skill\n\n## When to Use This Skill\n\nUse when deploying custom applications, troubleshooting slow systems, investigating service failures, optimizing workloads, managing users, configuring SSH, monitoring disk space, scheduling tasks, diagnosing network issues, or applying performance tuning.\n\n## Quick Start\n\n### Essential Commands\n\n**Service Management:**\n```bash\nsystemctl start nginx              # Start service\nsystemctl stop nginx               # Stop service\nsystemctl restart nginx            # Restart service\nsystemctl status nginx             # Check status\nsystemctl enable nginx             # Enable at boot\njournalctl -u nginx -f             # Follow service logs\n```\n\n**Process Monitoring:**\n```bash\ntop                                # Interactive process monitor\nhtop                               # Enhanced process monitor\nps aux | grep process_name         # Find specific process\nkill -15 PID                       # Graceful shutdown (SIGTERM)\nkill -9 PID                        # Force kill (SIGKILL)\n```\n\n**Disk Usage:**\n```bash\ndf -h                              # Filesystem usage\ndu -sh /path/to/dir                # Directory size\nncdu /path                         # Interactive disk analyzer\n```\n\n**Log Analysis:**\n```bash\njournalctl -f                      # Follow all logs\njournalctl -u service -f           # Follow service logs\njournalctl --since \"1 hour ago\"    # Filter by time\njournalctl -p err                  # Show errors only\n```\n\n**User Management:**\n```bash\nuseradd -m -s /bin/bash username   # Create user with home dir\npasswd username                    # Set password\nusermod -aG sudo username          # Add to sudo group\nuserdel -r username                # Delete user and home dir\n```\n\n## Core Concepts\n\n### Systemd Architecture\n\nSystemd is the standard init system and service manager. Systemd units define services, timers, targets, and other system resources.\n\n**Unit File Locations (priority order):**\n- `/etc/systemd/system/` - Custom units (highest priority)\n- `/run/systemd/system/` - Runtime units (transient)\n- `/lib/systemd/system/` - System-provided units (don't modify)\n\n**Key Unit Types:** `.service` (services), `.timer` (scheduled tasks), `.target` (unit groups), `.socket` (socket-activated)\n\n**Essential systemctl Commands:**\n```bash\nsystemctl daemon-reload            # Reload unit files after changes\nsystemctl list-units --type=service\nsystemctl list-timers              # Show all timers\nsystemctl cat nginx.service        # Show unit file content\nsystemctl edit nginx.service       # Create override file\n```\n\nFor detailed systemd reference, see `references/systemd-guide.md`.\n\n### Process Management\n\nProcesses are running programs with unique PIDs. Understanding process states, signals, and resource usage is essential for troubleshooting.\n\n**Process States:** R (running), S (sleeping), D (uninterruptible sleep/I/O), Z (zombie), T (stopped)\n\n**Common Signals:** SIGTERM (15) graceful, SIGKILL (9) force, SIGHUP (1) reload config\n\n**Process Priority:**\n```bash\nnice -n 10 command                 # Start with lower priority\nrenice -n 5 -p PID                 # Change priority of running process\n```\n\n### Filesystem Hierarchy\n\nEssential directories: `/` (root), `/etc/` (config), `/var/` (variable data), `/opt/` (optional software), `/usr/` (user programs), `/home/` (user directories), `/tmp/` (temporary), `/boot/` (boot loader)\n\n**Filesystem Types Quick Reference:**\n- **ext4** - General purpose (default)\n- **XFS** - Large files, databases (RHEL default)\n- **Btrfs** - Snapshots, copy-on-write\n- **ZFS** - Enterprise, data integrity, NAS\n\nFor filesystem management details including LVM and RAID, see `references/filesystem-management.md`.\n\n### Package Management\n\n**Ubuntu/Debian (apt):**\n```bash\napt update && apt upgrade          # Update system\napt install package                # Install package\napt remove package                 # Remove package\napt search keyword                 # Search packages\n```\n\n**RHEL/CentOS/Fedora (dnf):**\n```bash\ndnf update                         # Update all packages\ndnf install package                # Install package\ndnf remove package                 # Remove package\ndnf search keyword                 # Search packages\n```\n\nUse native package managers for system services; snap/flatpak for desktop apps and cross-distro compatibility.\n\n## Decision Frameworks\n\n### Troubleshooting Performance Issues\n\n**Investigation Workflow:**\n\n1. **Identify bottleneck:**\n   ```bash\n   top                             # Quick overview\n   uptime                          # Load averages\n   ```\n\n2. **CPU Issues (usage >80%):**\n   ```bash\n   top                             # Press Shift+P to sort by CPU\n   ps aux --sort=-%cpu | head\n   ```\n\n3. **Memory Issues (swap used):**\n   ```bash\n   free -h                         # Memory usage\n   top                             # Press Shift+M to sort by memory\n   ```\n\n4. **Disk I/O Issues (high wa%):**\n   ```bash\n   iostat -x 1                     # Disk statistics\n   iotop                           # I/O by process\n   ```\n\n5. **Network Issues:**\n   ```bash\n   ss -tunap                       # Active connections\n   iftop                           # Bandwidth monitor\n   ```\n\nFor comprehensive troubleshooting, see `references/troubleshooting-guide.md`.\n\n### Filesystem Selection\n\n**Quick Decision:**\n- **Default/General** → ext4\n- **Database servers** → XFS\n- **Large file storage** → XFS or ZFS\n- **NAS/File server** → ZFS\n- **Need snapshots** → Btrfs or ZFS\n\n## Common Workflows\n\n### Creating a Systemd Service\n\n**Step 1: Create unit file**\n```bash\nsudo nano /etc/systemd/system/myapp.service\n```\n\n**Step 2: Unit file content**\n```ini\n[Unit]\nDescription=My Web Application\nAfter=network.target postgresql.service\nRequires=postgresql.service\n\n[Service]\nType=simple\nUser=myapp\nGroup=myapp\nWorkingDirectory=/opt/myapp\nEnvironment=\"PORT=8080\"\nExecStart=/opt/myapp/bin/server\nExecReload=/bin/kill -HUP $MAINPID\nRestart=on-failure\nRestartSec=5s\nStandardOutput=journal\n\n# Security hardening\nPrivateTmp=true\nNoNewPrivileges=true\nProtectSystem=strict\nReadWritePaths=/var/lib/myapp\n\n[Install]\nWantedBy=multi-user.target\n```\n\n**Step 3: Deploy and start**\n```bash\nsudo useradd -r -s /bin/false myapp\nsudo mkdir -p /var/lib/myapp\nsudo chown myapp:myapp /var/lib/myapp\nsudo systemctl daemon-reload\nsudo systemctl enable myapp.service\nsudo systemctl start myapp.service\nsudo systemctl status myapp.service\n```\n\nFor complete examples, see `examples/systemd-units/`.\n\n### Systemd Timer (Cron Replacement)\n\nCreate service and timer units for scheduled tasks. Timer unit specifies `OnCalendar=` schedule and `Persistent=true` for missed jobs. Service unit has `Type=oneshot`. See `examples/systemd-units/backup.timer` and `backup.service` for complete examples.\n\n### SSH Hardening\n\n**Generate SSH key:**\n```bash\nssh-keygen -t ed25519 -C \"admin@example.com\"\nssh-copy-id admin@server\n```\n\n**Harden sshd_config:**\n```bash\nsudo nano /etc/ssh/sshd_config\n```\n\nKey settings:\n```bash\nPermitRootLogin no\nPasswordAuthentication no\nPubkeyAuthentication yes\nMaxAuthTries 3\nAllowUsers admin deploy\nX11Forwarding no\nPort 2222                          # Optional\n```\n\n**Apply changes:**\n```bash\nsudo sshd -t                       # Test\nsudo systemctl restart sshd        # Apply (keep backup session!)\n```\n\nFor complete SSH configuration, see `examples/configs/sshd_config.hardened` and `references/security-hardening.md`.\n\n### Performance Tuning\n\nConfigure sysctl parameters in `/etc/sysctl.d/99-custom.conf` for network tuning (tcp buffers, BBR congestion control), memory management (swappiness, cache pressure), and file descriptors. Set ulimits in `/etc/security/limits.conf` for nofile and nproc. Configure I/O schedulers and CPU governors. For comprehensive tuning, see `references/performance-tuning.md` and `examples/configs/` for templates.\n\n### Log Investigation\n\nUse `systemctl status myapp` and `journalctl -u myapp` to investigate issues. Filter logs by time `--since`, severity `-p err`, or search patterns with `grep`. Correlate with system metrics using `top`, `df -h`, `free -h`. Check for OOM kills with `journalctl -k | grep -i oom`. For detailed workflows, see `references/troubleshooting-guide.md`.\n\n### Essential Commands\n\n**Interface Management:**\n```bash\nip addr show                       # Show all interfaces\nip link set eth0 up                # Bring interface up\nip addr add 192.168.1.100/24 dev eth0\n```\n\n**Routing:**\n```bash\nip route show                      # Show routing table\nip route get 8.8.8.8               # Show route to IP\nip route add 10.0.0.0/24 via 192.168.1.1\n```\n\n**Socket Statistics:**\n```bash\nss -tunap                          # All TCP/UDP connections\nss -tlnp                           # Listening TCP ports\nss -ulnp                           # Listening UDP ports\nss -tnp state established          # Established connections\n```\n\n### Firewall Configuration\n\n**Ubuntu (ufw):**\n```bash\nsudo ufw status\nsudo ufw enable\nsudo ufw allow 22/tcp              # Allow SSH\nsudo ufw allow 80/tcp              # Allow HTTP\nsudo ufw allow from 192.168.1.0/24 # Allow from subnet\nsudo ufw default deny incoming\n```\n\n**RHEL/CentOS (firewalld):**\n```bash\nfirewall-cmd --state\nfirewall-cmd --list-all\nfirewall-cmd --add-service=http --permanent\nfirewall-cmd --add-port=8080/tcp --permanent\nfirewall-cmd --reload\n```\n\nFor complete network configuration including netplan, NetworkManager, and DNS, see `references/network-configuration.md`.\n\n## Scheduled Tasks\n\n### Cron Syntax\n\n```bash\ncrontab -e                         # Edit user crontab\n\n# Format: minute hour day month weekday command\n0 2 * * * /usr/local/bin/backup.sh              # Daily at 2:00 AM\n*/5 * * * * /usr/local/bin/check-health.sh      # Every 5 minutes\n0 3 * * 0 /usr/local/bin/weekly-cleanup.sh      # Weekly Sunday 3 AM\n@reboot /usr/local/bin/startup-script.sh        # Run at boot\n```\n\n### Systemd Timer Calendar Syntax\n\n```bash\nOnCalendar=daily                   # Every day at midnight\nOnCalendar=*-*-* 02:00:00          # Daily at 2:00 AM\nOnCalendar=Mon *-*-* 09:00:00      # Every Monday at 9 AM\nOnCalendar=*-*-01 00:00:00         # 1st of every month\nOnBootSec=5min                     # 5 minutes after boot\n```\n\n## Essential Tools\n\n### Process Monitoring\n- `top`, `htop` - Real-time process monitor\n- `ps` - Report process status\n- `pgrep/pkill` - Find/kill by name\n\n### Log Analysis\n- `journalctl` - Query systemd journal\n- `grep` - Search text patterns\n- `tail -f` - Follow log files\n\n### Disk Management\n- `df` - Disk space usage\n- `du` - Directory space usage\n- `lsblk` - List block devices\n- `ncdu` - Interactive disk analyzer\n\n### Network Tools\n- `ip` - Network configuration\n- `ss` - Socket statistics\n- `ping` - Test connectivity\n- `dig/nslookup` - DNS queries\n- `tcpdump` - Packet capture\n\n### System Monitoring\n- **Netdata** - Real-time web dashboard\n- **Prometheus + Grafana** - Metrics collection\n- **ELK Stack** - Centralized logging\n\n## Integration with Other Skills\n\n### Kubernetes Operations\nLinux administration is the foundation for Kubernetes node management. Node optimization (sysctl tuning), kubelet as systemd service, container logs via journald, cgroups for resource limits.\n\nExample:\n```bash\n# /etc/sysctl.d/99-kubernetes.conf\nnet.bridge.bridge-nf-call-iptables = 1\nnet.ipv4.ip_forward = 1\n```\n\nFor Kubernetes-specific operations, see `kubernetes-operations` skill.\n\n### Configuration Management\nLinux administration provides knowledge; configuration management automates it. Ansible playbooks automate systemd service creation and system tuning.\n\nFor automation at scale, see `configuration-management` skill.\n\n### Security Hardening\nThis skill covers SSH and firewall basics. For advanced security (MFA, certificates, CIS benchmarks, compliance), see `security-hardening` skill.\n\n### CI/CD Pipelines\nCI/CD pipelines deploy to Linux servers using these skills. Uses systemctl for deployment and journalctl for monitoring.\n\nFor deployment automation, see `building-ci-pipelines` skill.\n\n## Reference Materials\n\n### Detailed Guides\n- **`references/systemd-guide.md`** - Comprehensive systemd reference (unit files, dependencies, targets)\n- **`references/performance-tuning.md`** - Complete sysctl, ulimits, cgroups, I/O scheduler guide\n- **`references/filesystem-management.md`** - LVM, RAID, filesystem types, permissions\n- **`references/network-configuration.md`** - ip/ss commands, netplan, NetworkManager, DNS, firewall\n- **`references/security-hardening.md`** - SSH hardening, firewall, SELinux/AppArmor basics\n- **`references/troubleshooting-guide.md`** - Common issues, diagnostic workflows, solutions\n\n### Examples\n- **`examples/systemd-units/`** - Service, timer, and target unit files\n- **`examples/scripts/`** - Backup, health check, and maintenance scripts\n- **`examples/configs/`** - sshd_config, sysctl.conf, logrotate examples\n\n## Distribution-Specific Notes\n\n### Ubuntu/Debian\nPackage Manager: `apt`, Network: `netplan`, Firewall: `ufw`, Repositories: `/etc/apt/sources.list`\n\n### RHEL/CentOS/Fedora\nPackage Manager: `dnf`, Network: `NetworkManager`, Firewall: `firewalld`, Repositories: `/etc/yum.repos.d/`, SELinux enabled by default\n\n### Arch Linux\nPackage Manager: `pacman`, Network: `NetworkManager`, Rolling release, AUR for community packages\n\n## Additional Resources\n\n**Official Documentation:**\n- systemd: https://systemd.io/\n- Linux kernel: https://kernel.org/doc/\n\n**Related Skills:**\n- `kubernetes-operations` - Container orchestration on Linux\n- `configuration-management` - Automate Linux admin at scale\n- `security-hardening` - Advanced security and compliance\n- `building-ci-pipelines` - Deploy via CI/CD\n- `performance-engineering` - Deep performance analysis","author":"@ancoleman","ownerProfile":null,"authorContacts":null,"sourceUrl":"https://github.com/ancoleman/ai-design-components/tree/main/skills/administering-linux","license":"MIT","category":"devops","lang":"en","tokens":3547,"stars":0,"calls30d":2,"claimed":false,"visibility":"public","origin":"crawler","version":"0.1.0","createdAt":"2026-08-22","updatedAt":"2026-08-22","files":[{"path":"examples/configs/sshd_config.hardened","size":1874,"sha256":"b4f540e8f0e38a5183ade86c97732a920d4c25dc92c5c96d581eaea9ad3513c1"},{"path":"examples/scripts/backup.sh","size":2228,"sha256":"49c13c58e434cf2ab1a8ba43dc2fca0c5367b1398fe9996ce375a2ebf6c9bfd7"},{"path":"examples/systemd-units/backup.service","size":702,"sha256":"8591599a0dc9ca3934e3d125b069d542247b9bbd2cc5882ab755e7b219738602"},{"path":"examples/systemd-units/backup.timer","size":759,"sha256":"07be6cbfcb9d9080f1a8c2816c203a31c82ba3ad790916d1ed5d4137136b7be6"},{"path":"examples/systemd-units/webapp.service","size":1433,"sha256":"a22f7a4c4e69ae4352b13e785210f62ad9a533b66af98bcdfa25f072e08f0475"},{"path":"outputs.yaml","size":6025,"sha256":"1e947b43ddd8c6348977c1b77acd94661616126d80975f5a2d6bcb333fa7e262"},{"path":"references/filesystem-management.md","size":10464,"sha256":"d1770d499cdae66ef92ac63b2e120f951247fd839c7efe57e72636f81b6dccac"},{"path":"references/network-configuration.md","size":11341,"sha256":"1c7728434845139ee84b5adeeb9a1efacc28bb2389ff8c7f1edc6ddc22d27714"},{"path":"references/performance-tuning.md","size":20918,"sha256":"481aa9706407a8b00fdb3ded897ff503ea09d4cb952f8642dc5da38d69d9e20e"},{"path":"references/security-hardening.md","size":12959,"sha256":"62d7ed30e12d1fb8ee1bc7e3b40d54c3d577171b882984cc2678fbfe967e31e8"},{"path":"references/systemd-guide.md","size":19823,"sha256":"ff9d02c4b060c2a4b0e1ef5969b23ba052c31c3b4ee10e7adf220914510ac684"},{"path":"references/troubleshooting-guide.md","size":17652,"sha256":"4d73915252230dcd18ee9b7455eaf469bbc5fa605409d442d6677225d5a5f751"}],"requires":{"mcp":[],"tools":[]},"safety":{"flags":[{"code":"code.chmod-777","kind":"dangerous-code","where":"references/filesystem-management.md:334","excerpt":"chmod 777","message":"sets world-writable permissions","severity":"warn"},{"code":"code.eval","kind":"dangerous-code","where":"references/systemd-guide.md:932","excerpt":"exec(","message":"evaluates code at runtime","severity":"warn"},{"code":"net.endpoints","kind":"exfiltration","excerpt":"csrc.nist.gov, docs.example.com, firewalld.org, help.ubuntu.com, kernel.org, selinuxproject.org, systemd.io, wiki.ubuntu.com","message":"bundled scripts reach 10 external host(s)","severity":"warn"}],"scannedAt":"2026-08-22","hasScripts":true,"networkEndpoints":["csrc.nist.gov","docs.example.com","firewalld.org","help.ubuntu.com","kernel.org","selinuxproject.org","systemd.io","wiki.ubuntu.com","www.cisecurity.org","www.kernel.org"]}}