{"id":"spark-environment-setup","name":"spark-environment-setup","summary":"NVIDIA DGX Spark(GB10、aarch64、CUDA 13)上で動作するMLトレーニング/推論環境を構築します。","body":"# Spark Environment Setup\n\nDGX Spark ships a GB10 Grace Blackwell chip: aarch64 CPU, SM121\nGPU, 128GB unified memory, CUDA 13. This is a narrower and\nyounger platform than a standard x86 CUDA 12 box, so package\nselection and ABI matching matter more than usual — the wheel\necosystem for aarch64 + CUDA 13 is still filling in.\n\n## When to Use This Skill\n\n- Setting up a fresh Spark box for training or inference.\n- Hitting an import error mentioning `libcudart`, a missing\n  symbol, or a wheel that \"installed fine but won't load.\"\n- A framework install (PyTorch, Unsloth, TRL, vLLM, xformers)\n  fails, hangs, or silently falls back to CPU.\n- Deciding whether to use an NGC container or bare pip.\n- Restoring a working setup after an OS reinstall or a\n  base-image update, needing to re-verify from scratch.\n\nEach of these accepts the same general fix: match the\ncontainer/wheel combination to CUDA 13 and SM121, don't fight\nthe ABI.\n\n## Container-First Rule\n\nQuick decision, before the detail below:\n\n- Standard training/inference work → NGC PyTorch container.\n- Unsloth-centric fine-tuning → Unsloth container (it ships\n  the pinned Triton/xformers/transformers combination already\n  validated for that path).\n- Neither fits (custom system package, local IDE interpreter)\n  → bare pip, following the exact sequence further down.\n\nDefault to a container. Use `nvcr.io/nvidia/pytorch:25.09-py3`\nas the base for general work — the newest tag confirmed working\non this hardware; pull a newer blessed tag if locally available\nrather than hard-blocking on `25.11-py3`. NGC's tag is dated, so\nrunning it directly is fine:\n\n```bash\ndocker run --runtime=nvidia --gpus all -it --rm \\\n  nvcr.io/nvidia/pytorch:25.09-py3\n```\n\n`unsloth/unsloth:dgxspark-latest` is a *moving* tag by\ncontrast — resolve and pin its digest before running it for\nanything reproducible; the bare tag is a discovery step only,\nnot the default invocation. Full pull-inspect-pin sequence and\nflag rationale/volume mounts for `finetuning/` run dirs:\n`references/container-workflow.md`. Treat bare pip as the exception.\n\nThe reason for the container-first stance is pinning, not\nconvenience. Triton, xformers, and transformers versions\ninteract narrowly with GB10's SM121 target and CUDA 13; a\ncontainer locks all of them together against a combination\nalready validated on this hardware. Bare pip leaves that\nresolution to you, one broken import at a time.\n\nWhen bare pip is warranted, follow the NVIDIA playbook's\ninstall sequence verbatim and in order:\n\n```bash\npip install \"transformers==5.13.1\" \"peft==0.19.1\" \"hf_transfer==0.1.9\" \"datasets==4.3.0\" \"trl==1.8.0\"\npip install --no-deps \"unsloth==2026.7.2\" \"unsloth_zoo==2026.7.2\" \"bitsandbytes==0.49.2\"\npip install -U \"torchao==0.17.0\"\n```\n\nThe second command's `--no-deps` flag is not optional —\nletting pip re-resolve Unsloth's dependency tree on aarch64 is\na common way to pull in an incompatible torch or triton build.\nThe third line is not optional either: the NGC base image's\nbundled `torchao` is too old for current `peft`'s LoRA-attach\npath (`ImportError: ... torchao ... only versions above 0.16.0\nare supported`) — a hard blocker, not a warning. Every `==` pin\nabove is load-bearing, taken from the dated known-good version\nmatrix in `references/stack-matrix.md` (its `Last verified` date\ngoverns staleness) — an unpinned install resolves current PyPI\nversions well outside what this Unsloth release supports.\n\nPull a fresh tag when a new blessed release is announced.\nRebuild locally from one of the two bases only when a project\nneeds an extra system package layered in — not to \"upgrade\" a\ncomponent the image already pins. Details on both paths:\n`references/container-workflow.md`.\n\nOne more preflight: official DGX Spark playbooks have shipped\nbroken before. Check recent issues on\n`github.com/NVIDIA/dgx-spark-playbooks` (and the other\nresources in `references/stack-matrix.md`) before trusting a\nrecipe verbatim for a long run.\n\n## The ABI Rule\n\nThe single most common failure on Spark is a CUDA 12/13 ABI\nmismatch: a wheel built against `libcudart.so.12` loaded on a\nsystem that only has `libcudart.so.13`. The install usually\nsucceeds; the failure surfaces later as a missing-symbol error\nor a segfault that doesn't obviously point at CUDA.\n\nFix: pull wheels from `download.pytorch.org/whl/cu130` (the\ncu130-tagged aarch64 builds), or use one of the containers\nabove, which already carry a matched build. Before chasing a\nstack trace that mentions a CUDA symbol, check which CUDA tag\nthe installed wheel was built against:\n\n```bash\npython3 -c \"import torch; print(torch.version.cuda)\"\n```\n\nIf that output doesn't start with `13`, the ABI mismatch is the\nfirst thing to fix. NGC container builds (e.g.\n`nvcr.io/nvidia/pytorch:25.09-py3`) build torch internally\nagainst CUDA 13 with no `+cu130` wheel tag — `pip show torch`\nwon't say `cu130` there, and that absence alone is not a failure.\n\nTypical symptoms:\n\n- `ImportError: undefined symbol` referencing a CUDA runtime\n  function.\n- A segfault on the first `.cuda()` call, no useful traceback.\n- A wheel that installs cleanly, then fails at import time —\n  pip's resolver doesn't check CUDA ABI, only version constraints.\n- Two \"identical\" environments behaving differently — usually one\n  has a cu130 wheel, the other a cu121/cu124 leftover.\n\nThe fix is the same regardless of symptom: match the wheel's\nCUDA tag to the system, or use a container that already does.\n\n## Component Quick Table\n\nCondensed status for the components most likely to come up.\nFull table with wheel URLs, build flags, the sm_121 vs sm_121a\ndistinction, and the dated known-good version matrix:\n`references/stack-matrix.md`.\n\n| Component | Status |\n|---|---|\n| PyTorch | ✅ official cu130 aarch64 wheels |\n| bitsandbytes | ✅ works out of the box |\n| Triton | ✅ needs the `TRITON_PTXAS_PATH` parameter set |\n| flash-attn | ❌ skip pip build; NGC bundles a working one — see `spark-training-gotchas` G2 |\n| xformers | source build only (`TORCH_CUDA_ARCH_LIST=12.1`) |\n| vLLM | nightly wheels only |\n| TransformerEngine / NVFP4 train | container-only |\n\nEverything else — Unsloth, Axolotl, TRL, PEFT — installs\ncleanly through the container-first path above. LLaMA-Factory\nand NeMo are fragile on Spark; check upstream issues first.\n\n## Verification Commands\n\nConfirm the environment can actually see the GPU before\nrunning anything expensive:\n\n```python\nimport torch\nprint(torch.cuda.is_available(), torch.version.cuda)\n```\n\nThis call returns two values; the exact output format is one\nline, `<bool> <cuda-version>`:\n\n```text\nTrue 13.0\n```\n\nIf it prints `False` instead, don't jump straight to a wheel\nreinstall — ABI mismatch is one cause among several:\n\n| Hypothesis | Quick check |\n|---|---|\n| Runtime/flags | `nvidia-smi` fails in-container too |\n| Device visibility | `echo $CUDA_VISIBLE_DEVICES` |\n| Permissions | `ls -l /dev/nvidia*` |\n| CUDA init state | wedged process; retry fresh shell/container |\n| ABI mismatch (usual culprit) | `torch.version.cuda` not `13.x` |\n\nCheck `nvidia-smi` first — if it doesn't show the GPU, it's one\nof the first three, not ABI. Reinstall a wheel only once ABI is\nconfirmed. Per-hypothesis detail: `references/stack-matrix.md`.\nRun right after the container starts, before installing\nproject-specific packages.\n\nOne more check: if Triton kernel compilation fails once\ntraining starts, set\n`TRITON_PTXAS_PATH=/usr/local/cuda/bin/ptxas` and retry — see\n`references/stack-matrix.md` for the full workaround list.\n\n## Next Steps\n\nA verified environment is only the starting point. See also:\n`spark-training-gotchas` for failure preflights before a\ntraining run, and `spark-memory-thermal-ops` for unified-memory\nOOMs and thermal throttling during long ones.","author":"@wshobson","ownerProfile":null,"authorContacts":null,"sourceUrl":"https://github.com/wshobson/agents/tree/main/plugins/dgx-spark-ops/skills/spark-environment-setup","license":"MIT","category":null,"lang":"en","tokens":2012,"stars":0,"calls30d":2,"claimed":false,"visibility":"public","origin":"crawler","version":"0.1.0","createdAt":"2026-08-22","updatedAt":"2026-08-22","files":[{"path":"references/container-workflow.md","size":4171,"sha256":"1daaf8bfc02b00e054d8107fac9c3ad367a58529f1361f3ae2a329d0f0434ccc"},{"path":"references/stack-matrix.md","size":7064,"sha256":"8e9fccea72e8ffba0b143f01d159e100485ce1838d1e42c02221fa9cd1ed2980"}],"requires":{"mcp":[],"tools":[]},"safety":{"flags":[],"scannedAt":"2026-08-22","hasScripts":false,"networkEndpoints":[]}}