GraphTok

Website and docs: https://msiavashi.github.io/GraphTok/

GraphTok is a BPE tokenizer that runs end to end on an NVIDIA GPU. Pre-tokenization, BPE merging and output assembly are captured as one replayable CUDA graph, and token IDs are bit-identical to the reference tokenizers (tiktoken for GPT-2, Hugging Face tokenizers for the rest). Small inputs can be routed to a CPU engine, and a load-adaptive dispatcher batches concurrent requests into single graph replays. Repository: https://github.com/Msiavashi/GraphTok.

• Exact: token IDs match the reference tokenizer bit for bit.
• Built for long contexts: encodes 100k to 1M-token prompts in one graph replay.
• Drop-in for serving: a Python module, a C library, a vLLM plugin and an NVIDIA Dynamo backend.

Names: Python module gpu_bpe_tokenizer, C library libgtok, CLI gpu_bpe_tokenize.

News

• 2026-10 — v0.3.0, the first public release: GPU tokenizer, Python bindings, libgtok C ABI, vLLM plugin and NVIDIA Dynamo backend patch.
• 2026-10 — Paper "Taking Tokenization off the Host for Agentic LLM Serving" on arXiv (arXiv link coming soon).

Supported tokenizers

The all preset (and pip install .) builds GPT-2, Llama 3, DeepSeek-V3 and Gemma 3. Build Qwen with the qwen25 preset. See docs/BUILD.md for every preset and option.

Quick start

Requirements: Linux, an NVIDIA GPU, CUDA toolkit 12.6 or newer (tested with 12.6, 12.8 and 13.1), CMake >= 3.24, a C++17 compiler, Python >= 3.10 (utf8proc when Qwen is enabled).

git clone https://github.com/Msiavashi/GraphTok.git && cd GraphTok

# Python module
pip install .                                   # or '.[test]' / '.[vllm]'

# CLI and libgtok
cmake --preset all && cmake --build build/all -j

# Vocabulary files (Llama 3 and Gemma 3 need a token with accepted licenses)
HF_TOKEN=hf_... python3 scripts/download-vocabs.py

Encode a file with the CLI:

./build/all/gpu_bpe_tokenize --vocab data/hf_gpt2_tokenizer.json \
    --input prompt.txt --output tokens.bin

Encode in Python:

import gpu_bpe_tokenizer as gbpe

tok = gbpe.Tokenizer("data/hf_gpt2_tokenizer.json")
ids = tok.encode("Hello, world!")             # list[int]
text = tok.decode(ids)

tokens, offsets = tok.encode_batch(["first doc", "second doc"])
doc0 = tokens[offsets[0]:offsets[1]]

# thread-safe, load-adaptive encoder for many concurrent callers
d = gbpe.Dispatcher("data/hf_gpt2_tokenizer.json", cpu_workers=2)
arr = d.encode_numpy("Hello, world!")

Useful commands

The Makefile wraps the common workflows; make help lists them:

Integrations

vLLM. The package registers a vllm.general_plugins entry point that routes string encode / __call__ through libgtok.so.

pip install '.[vllm]'
cmake --preset qwen25 && cmake --build build/qwen25 -j --target gtok   # builds libgtok.so
GBPE_VLLM=1 GBPE_GTOK_LIB=$PWD/build/qwen25/libgtok.so vllm serve Qwen/Qwen3-32B

NVIDIA Dynamo. See integrations/dynamo/README.md.

Development

Development happens on the develop branch and pull requests target develop; main holds releases. See docs/DEVELOPMENT.md for the environment, tests and release process, and CONTRIBUTING.md before opening a pull request.

Documentation

• docs/BUILD.md — build presets and options
• docs/ARCHITECTURE.md — design of the GPU pipeline
• docs/ADDING_A_TOKENIZER.md — adding a vocabulary
• docs/DEVELOPMENT.md — development workflow
• CHANGELOG.md — release history

Citation

@misc{siavashi2026graphtok,
  title={Taking Tokenization off the Host for Agentic {LLM} Serving},
  author={Siavashi, Mohammad and Schwinte, Pascal and Banaei, Ali and Maguire Jr., Gerald Q. and Chiesa, Marco and Kosti{\'c}, Dejan},
  year={2026},
  note={arXiv preprint}
}

A machine-readable CITATION.cff is also provided.

License

Apache License 2.0. See LICENSE and NOTICE.