{"id":"huggingface-tokenizers","name":"huggingface-tokenizers","summary":"研究と生産に最適化された高速トークナイザー。Rustベースの実装は1GBを<20秒でトークン化します。","body":"# HuggingFace Tokenizers - Fast Tokenization for NLP\n\nFast, production-ready tokenizers with Rust performance and Python ease-of-use.\n\n## When to use HuggingFace Tokenizers\n\n**Use HuggingFace Tokenizers when:**\n- Need extremely fast tokenization (<20s per GB of text)\n- Training custom tokenizers from scratch\n- Want alignment tracking (token → original text position)\n- Building production NLP pipelines\n- Need to tokenize large corpora efficiently\n\n**Performance**:\n- **Speed**: <20 seconds to tokenize 1GB on CPU\n- **Implementation**: Rust core with Python/Node.js bindings\n- **Efficiency**: 10-100× faster than pure Python implementations\n\n**Use alternatives instead**:\n- **SentencePiece**: Language-independent, used by T5/ALBERT\n- **tiktoken**: OpenAI's BPE tokenizer for GPT models\n- **transformers AutoTokenizer**: Loading pretrained only (uses this library internally)\n\n## Quick start\n\n### Installation\n\n```bash\n# Install tokenizers\npip install tokenizers\n\n# With transformers integration\npip install tokenizers transformers\n```\n\n### Load pretrained tokenizer\n\n```python\nfrom tokenizers import Tokenizer\n\n# Load from HuggingFace Hub\ntokenizer = Tokenizer.from_pretrained(\"bert-base-uncased\")\n\n# Encode text\noutput = tokenizer.encode(\"Hello, how are you?\")\nprint(output.tokens)  # ['hello', ',', 'how', 'are', 'you', '?']\nprint(output.ids)     # [7592, 1010, 2129, 2024, 2017, 1029]\n\n# Decode back\ntext = tokenizer.decode(output.ids)\nprint(text)  # \"hello, how are you?\"\n```\n\n### Train custom BPE tokenizer\n\n```python\nfrom tokenizers import Tokenizer\nfrom tokenizers.models import BPE\nfrom tokenizers.trainers import BpeTrainer\nfrom tokenizers.pre_tokenizers import Whitespace\n\n# Initialize tokenizer with BPE model\ntokenizer = Tokenizer(BPE(unk_token=\"[UNK]\"))\ntokenizer.pre_tokenizer = Whitespace()\n\n# Configure trainer\ntrainer = BpeTrainer(\n    vocab_size=30000,\n    special_tokens=[\"[UNK]\", \"[CLS]\", \"[SEP]\", \"[PAD]\", \"[MASK]\"],\n    min_frequency=2\n)\n\n# Train on files\nfiles = [\"train.txt\", \"validation.txt\"]\ntokenizer.train(files, trainer)\n\n# Save\ntokenizer.save(\"my-tokenizer.json\")\n```\n\n**Training time**: ~1-2 minutes for 100MB corpus, ~10-20 minutes for 1GB\n\n### Batch encoding with padding\n\n```python\n# Enable padding\ntokenizer.enable_padding(pad_id=3, pad_token=\"[PAD]\")\n\n# Encode batch\ntexts = [\"Hello world\", \"This is a longer sentence\"]\nencodings = tokenizer.encode_batch(texts)\n\nfor encoding in encodings:\n    print(encoding.ids)\n# [101, 7592, 2088, 102, 3, 3, 3]\n# [101, 2023, 2003, 1037, 2936, 6251, 102]\n```\n\n## Tokenization algorithms\n\n### BPE (Byte-Pair Encoding)\n\n**How it works**:\n1. Start with character-level vocabulary\n2. Find most frequent character pair\n3. Merge into new token, add to vocabulary\n4. Repeat until vocabulary size reached\n\n**Used by**: GPT-2, GPT-3, RoBERTa, BART, DeBERTa\n\n```python\nfrom tokenizers import Tokenizer\nfrom tokenizers.models import BPE\nfrom tokenizers.trainers import BpeTrainer\nfrom tokenizers.pre_tokenizers import ByteLevel\n\ntokenizer = Tokenizer(BPE(unk_token=\"<|endoftext|>\"))\ntokenizer.pre_tokenizer = ByteLevel()\n\ntrainer = BpeTrainer(\n    vocab_size=50257,\n    special_tokens=[\"<|endoftext|>\"],\n    min_frequency=2\n)\n\ntokenizer.train(files=[\"data.txt\"], trainer=trainer)\n```\n\n**Advantages**:\n- Handles OOV words well (breaks into subwords)\n- Flexible vocabulary size\n- Good for morphologically rich languages\n\n**Trade-offs**:\n- Tokenization depends on merge order\n- May split common words unexpectedly\n\n### WordPiece\n\n**How it works**:\n1. Start with character vocabulary\n2. Score merge pairs: `frequency(pair) / (frequency(first) × frequency(second))`\n3. Merge highest scoring pair\n4. Repeat until vocabulary size reached\n\n**Used by**: BERT, DistilBERT, MobileBERT\n\n```python\nfrom tokenizers import Tokenizer\nfrom tokenizers.models import WordPiece\nfrom tokenizers.trainers import WordPieceTrainer\nfrom tokenizers.pre_tokenizers import Whitespace\nfrom tokenizers.normalizers import BertNormalizer\n\ntokenizer = Tokenizer(WordPiece(unk_token=\"[UNK]\"))\ntokenizer.normalizer = BertNormalizer(lowercase=True)\ntokenizer.pre_tokenizer = Whitespace()\n\ntrainer = WordPieceTrainer(\n    vocab_size=30522,\n    special_tokens=[\"[UNK]\", \"[CLS]\", \"[SEP]\", \"[PAD]\", \"[MASK]\"],\n    continuing_subword_prefix=\"##\"\n)\n\ntokenizer.train(files=[\"corpus.txt\"], trainer=trainer)\n```\n\n**Advantages**:\n- Prioritizes meaningful merges (high score = semantically related)\n- Used successfully in BERT (state-of-the-art results)\n\n**Trade-offs**:\n- Unknown words become `[UNK]` if no subword match\n- Saves vocabulary, not merge rules (larger files)\n\n### Unigram\n\n**How it works**:\n1. Start with large vocabulary (all substrings)\n2. Compute loss for corpus with current vocabulary\n3. Remove tokens with minimal impact on loss\n4. Repeat until vocabulary size reached\n\n**Used by**: ALBERT, T5, mBART, XLNet (via SentencePiece)\n\n```python\nfrom tokenizers import Tokenizer\nfrom tokenizers.models import Unigram\nfrom tokenizers.trainers import UnigramTrainer\n\ntokenizer = Tokenizer(Unigram())\n\ntrainer = UnigramTrainer(\n    vocab_size=8000,\n    special_tokens=[\"<unk>\", \"<s>\", \"</s>\"],\n    unk_token=\"<unk>\"\n)\n\ntokenizer.train(files=[\"data.txt\"], trainer=trainer)\n```\n\n**Advantages**:\n- Probabilistic (finds most likely tokenization)\n- Works well for languages without word boundaries\n- Handles diverse linguistic contexts\n\n**Trade-offs**:\n- Computationally expensive to train\n- More hyperparameters to tune\n\n## Tokenization pipeline\n\nComplete pipeline: **Normalization → Pre-tokenization → Model → Post-processing**\n\n### Normalization\n\nClean and standardize text:\n\n```python\nfrom tokenizers.normalizers import NFD, StripAccents, Lowercase, Sequence\n\ntokenizer.normalizer = Sequence([\n    NFD(),           # Unicode normalization (decompose)\n    Lowercase(),     # Convert to lowercase\n    StripAccents()   # Remove accents\n])\n\n# Input: \"Héllo WORLD\"\n# After normalization: \"hello world\"\n```\n\n**Common normalizers**:\n- `NFD`, `NFC`, `NFKD`, `NFKC` - Unicode normalization forms\n- `Lowercase()` - Convert to lowercase\n- `StripAccents()` - Remove accents (é → e)\n- `Strip()` - Remove whitespace\n- `Replace(pattern, content)` - Regex replacement\n\n### Pre-tokenization\n\nSplit text into word-like units:\n\n```python\nfrom tokenizers.pre_tokenizers import Whitespace, Punctuation, Sequence, ByteLevel\n\n# Split on whitespace and punctuation\ntokenizer.pre_tokenizer = Sequence([\n    Whitespace(),\n    Punctuation()\n])\n\n# Input: \"Hello, world!\"\n# After pre-tokenization: [\"Hello\", \",\", \"world\", \"!\"]\n```\n\n**Common pre-tokenizers**:\n- `Whitespace()` - Split on spaces, tabs, newlines\n- `ByteLevel()` - GPT-2 style byte-level splitting\n- `Punctuation()` - Isolate punctuation\n- `Digits(individual_digits=True)` - Split digits individually\n- `Metaspace()` - Replace spaces with ▁ (SentencePiece style)\n\n### Post-processing\n\nAdd special tokens for model input:\n\n```python\nfrom tokenizers.processors import TemplateProcessing\n\n# BERT-style: [CLS] sentence [SEP]\ntokenizer.post_processor = TemplateProcessing(\n    single=\"[CLS] $A [SEP]\",\n    pair=\"[CLS] $A [SEP] $B [SEP]\",\n    special_tokens=[\n        (\"[CLS]\", 1),\n        (\"[SEP]\", 2),\n    ],\n)\n```\n\n**Common patterns**:\n```python\n# GPT-2: sentence <|endoftext|>\nTemplateProcessing(\n    single=\"$A <|endoftext|>\",\n    special_tokens=[(\"<|endoftext|>\", 50256)]\n)\n\n# RoBERTa: <s> sentence </s>\nTemplateProcessing(\n    single=\"<s> $A </s>\",\n    pair=\"<s> $A </s> </s> $B </s>\",\n    special_tokens=[(\"<s>\", 0), (\"</s>\", 2)]\n)\n```\n\n## Alignment tracking\n\nTrack token positions in original text:\n\n```python\noutput = tokenizer.encode(\"Hello, world!\")\n\n# Get token offsets\nfor token, offset in zip(output.tokens, output.offsets):\n    start, end = offset\n    print(f\"{token:10} → [{start:2}, {end:2}): {text[start:end]!r}\")\n\n# Output:\n# hello      → [ 0,  5): 'Hello'\n# ,          → [ 5,  6): ','\n# world      → [ 7, 12): 'world'\n# !          → [12, 13): '!'\n```\n\n**Use cases**:\n- Named entity recognition (map predictions back to text)\n- Question answering (extract answer spans)\n- Token classification (align labels to original positions)\n\n## Integration with transformers\n\n### Load with AutoTokenizer\n\n```python\nfrom transformers import AutoTokenizer\n\n# AutoTokenizer automatically uses fast tokenizers\ntokenizer = AutoTokenizer.from_pretrained(\"bert-base-uncased\")\n\n# Check if using fast tokenizer\nprint(tokenizer.is_fast)  # True\n\n# Access underlying tokenizers.Tokenizer\nfast_tokenizer = tokenizer.backend_tokenizer\nprint(type(fast_tokenizer))  # <class 'tokenizers.Tokenizer'>\n```\n\n### Convert custom tokenizer to transformers\n\n```python\nfrom tokenizers import Tokenizer\nfrom transformers import PreTrainedTokenizerFast\n\n# Train custom tokenizer\ntokenizer = Tokenizer(BPE())\n# ... train tokenizer ...\ntokenizer.save(\"my-tokenizer.json\")\n\n# Wrap for transformers\ntransformers_tokenizer = PreTrainedTokenizerFast(\n    tokenizer_file=\"my-tokenizer.json\",\n    unk_token=\"[UNK]\",\n    pad_token=\"[PAD]\",\n    cls_token=\"[CLS]\",\n    sep_token=\"[SEP]\",\n    mask_token=\"[MASK]\"\n)\n\n# Use like any transformers tokenizer\noutputs = transformers_tokenizer(\n    \"Hello world\",\n    padding=True,\n    truncation=True,\n    max_length=512,\n    return_tensors=\"pt\"\n)\n```\n\n## Common patterns\n\n### Train from iterator (large datasets)\n\n```python\nfrom datasets import load_dataset\n\n# Load dataset\ndataset = load_dataset(\"wikitext\", \"wikitext-103-raw-v1\", split=\"train\")\n\n# Create batch iterator\ndef batch_iterator(batch_size=1000):\n    for i in range(0, len(dataset), batch_size):\n        yield dataset[i:i + batch_size][\"text\"]\n\n# Train tokenizer\ntokenizer.train_from_iterator(\n    batch_iterator(),\n    trainer=trainer,\n    length=len(dataset)  # For progress bar\n)\n```\n\n**Performance**: Processes 1GB in ~10-20 minutes\n\n### Enable truncation and padding\n\n```python\n# Enable truncation\ntokenizer.enable_truncation(max_length=512)\n\n# Enable padding\ntokenizer.enable_padding(\n    pad_id=tokenizer.token_to_id(\"[PAD]\"),\n    pad_token=\"[PAD]\",\n    length=512  # Fixed length, or None for batch max\n)\n\n# Encode with both\noutput = tokenizer.encode(\"This is a long sentence that will be truncated...\")\nprint(len(output.ids))  # 512\n```\n\n### Multi-processing\n\n```python\nfrom tokenizers import Tokenizer\nfrom multiprocessing import Pool\n\n# Load tokenizer\ntokenizer = Tokenizer.from_file(\"tokenizer.json\")\n\ndef encode_batch(texts):\n    return tokenizer.encode_batch(texts)\n\n# Process large corpus in parallel\nwith Pool(8) as pool:\n    # Split corpus into chunks\n    chunk_size = 1000\n    chunks = [corpus[i:i+chunk_size] for i in range(0, len(corpus), chunk_size)]\n\n    # Encode in parallel\n    results = pool.map(encode_batch, chunks)\n```\n\n**Speedup**: 5-8× with 8 cores\n\n## Performance benchmarks\n\n### Training speed\n\n| Corpus Size | BPE (30k vocab) | WordPiece (30k) | Unigram (8k) |\n|-------------|-----------------|-----------------|--------------|\n| 10 MB       | 15 sec          | 18 sec          | 25 sec       |\n| 100 MB      | 1.5 min         | 2 min           | 4 min        |\n| 1 GB        | 15 min          | 20 min          | 40 min       |\n\n**Hardware**: 16-core CPU, tested on English Wikipedia\n\n### Tokenization speed\n\n| Implementation | 1 GB corpus | Throughput    |\n|----------------|-------------|---------------|\n| Pure Python    | ~20 minutes | ~50 MB/min    |\n| HF Tokenizers  | ~15 seconds | ~4 GB/min     |\n| **Speedup**    | **80×**     | **80×**       |\n\n**Test**: English text, average sentence length 20 words\n\n### Memory usage\n\n| Task                    | Memory  |\n|-------------------------|---------|\n| Load tokenizer          | ~10 MB  |\n| Train BPE (30k vocab)   | ~200 MB |\n| Encode 1M sentences     | ~500 MB |\n\n## Supported models\n\nPre-trained tokenizers available via `from_pretrained()`:\n\n**BERT family**:\n- `bert-base-uncased`, `bert-large-cased`\n- `distilbert-base-uncased`\n- `roberta-base`, `roberta-large`\n\n**GPT family**:\n- `gpt2`, `gpt2-medium`, `gpt2-large`\n- `distilgpt2`\n\n**T5 family**:\n- `t5-small`, `t5-base`, `t5-large`\n- `google/flan-t5-xxl`\n\n**Other**:\n- `facebook/bart-base`, `facebook/mbart-large-cc25`\n- `albert-base-v2`, `albert-xlarge-v2`\n- `xlm-roberta-base`, `xlm-roberta-large`\n\nBrowse all: https://huggingface.co/models?library=tokenizers\n\n## References\n\n- **[Training Guide](references/training.md)** - Train custom tokenizers, configure trainers, handle large datasets\n- **[Algorithms Deep Dive](references/algorithms.md)** - BPE, WordPiece, Unigram explained in detail\n- **[Pipeline Components](references/pipeline.md)** - Normalizers, pre-tokenizers, post-processors, decoders\n- **[Transformers Integration](references/integration.md)** - AutoTokenizer, PreTrainedTokenizerFast, special tokens\n\n## Resources\n\n- **Docs**: https://huggingface.co/docs/tokenizers\n- **GitHub**: https://github.com/huggingface/tokenizers ⭐ 9,000+\n- **Version**: 0.20.0+\n- **Course**: https://huggingface.co/learn/nlp-course/chapter6/1\n- **Paper**: BPE (Sennrich et al., 2016), WordPiece (Schuster & Nakajima, 2012)","author":"@Orchestra-Research","ownerProfile":null,"authorContacts":null,"sourceUrl":"https://github.com/Orchestra-Research/AI-Research-SKILLs/tree/main/02-tokenization/huggingface-tokenizers","license":"MIT","category":"coding","lang":"en","tokens":3421,"stars":0,"calls30d":2,"claimed":false,"visibility":"public","origin":"crawler","version":"0.1.0","createdAt":"2026-08-22","updatedAt":"2026-08-22","files":[{"path":"references/algorithms.md","size":15182,"sha256":"b26bb48dca45c57797e23ee8d3afa3c602ffbb243a47c0a9a171f3f6828a08ed"},{"path":"references/integration.md","size":15389,"sha256":"9e9c3f938424434da8e7b962ff536b2b3b4b585a0dabc5c9292a12c2a1682349"},{"path":"references/pipeline.md","size":16746,"sha256":"3fe8a70a27b1bf9aa52f6320a4da569c510fcf602e026cfc57a3f1518175cfc6"},{"path":"references/training.md","size":14559,"sha256":"9732225aacd7dacd83063a1c206e8566793fe41c628fd32f827e3489246dfcfe"}],"requires":{"mcp":[],"tools":[]},"safety":{"flags":[],"scannedAt":"2026-08-22","hasScripts":false,"networkEndpoints":["huggingface.co"]}}