v0.3.0 · Apache-2.0

GraphTok

Exact BPE tokenization, end to end on the GPU, as one replayable CUDA graph.

encode.py

import gpu_bpe_tokenizer as gbpe

tok = gbpe.Tokenizer("data/hf_gpt2_tokenizer.json")
ids = tok.encode("Hello, world!")             # list[int]
text = tok.decode(ids)

tokens, offsets = tok.encode_batch(["first doc", "second doc"])
doc0 = tokens[offsets[0]:offsets[1]]

Why GraphTok

A tokenizer whose hot path is entirely GPU work, built for the prompt sizes that agentic serving produces.

Bit-exact token IDs

Output matches tiktoken for GPT-2 and Hugging Face tokenizers for every other vocabulary, byte for byte, as a hard gate on every change.

Built for 100k to 1M tokens

Pre-token boundaries, merges and assembly are all parallel over the sequence, so a single long prompt is one replay rather than a scan.

One CUDA graph replay

Pre-allocated workspace, fixed shapes, no host synchronization inside the captured region. A call is one copy in, one replay, one copy out.

Drops into vLLM and Dynamo

A Python module, the libgtok C library, a vLLM general plugin and an NVIDIA Dynamo tokenizer backend share the same encoder.

How it works

Three stages, captured once per input-size class and replayed as a single graph.

01 · PRE-TOKENIZE

Boundaries as a parallel predicate

Each codepoint is classified on the GPU and a per-family boundary kernel decides is_boundary[k] from a bounded window. Prefix scans turn the flags into pre-token offsets.

02 · MERGE

Warp-cooperative BPE in reference order

One warp per pre-token looks up every adjacent pair in a packed merge hash, takes the minimum rank with __shfl_xor_sync and applies it, reproducing the reference merge order exactly.

03 · ASSEMBLE

Scan and flatten

An exclusive scan over per-pre-token counts gives write offsets, and the flatten kernels write the final IDs contiguously, with per-document offsets for batched encodes.

CPU route. Inputs at or below a content-aware byte threshold are encoded on the host by gigatoken or the built-in encoder, with the same IDs.

Dispatcher. A thread-safe encoder for many concurrent callers: small requests take a free CPU engine, everything else is batched into the next graph replay.

Quick start

Linux, an NVIDIA GPU, CUDA toolkit 12.6 or newer, CMake 3.24 or newer, a C++17 compiler and Python 3.10 or newer. The full build matrix is in the build guide.

Install

git clone https://github.com/Msiavashi/GraphTok.git && cd GraphTok

# Python module
pip install .                                   # or '.[test]' / '.[vllm]'

# CLI and libgtok
cmake --preset all && cmake --build build/all -j

# Vocabulary files (Llama 3 and Gemma 3 need a token with accepted licenses)
HF_TOKEN=hf_... python3 scripts/download-vocabs.py

CLI

./build/all/gpu_bpe_tokenize --vocab data/hf_gpt2_tokenizer.json \
    --input prompt.txt --output tokens.bin

Python

import gpu_bpe_tokenizer as gbpe

tok = gbpe.Tokenizer("data/hf_gpt2_tokenizer.json")
ids = tok.encode("Hello, world!")             # list[int]
text = tok.decode(ids)

tokens, offsets = tok.encode_batch(["first doc", "second doc"])
doc0 = tokens[offsets[0]:offsets[1]]

# thread-safe, load-adaptive encoder for many concurrent callers
d = gbpe.Dispatcher("data/hf_gpt2_tokenizer.json", cpu_workers=2)
arr = d.encode_numpy("Hello, world!")

vLLM

pip install '.[vllm]'
cmake --preset qwen25 && cmake --build build/qwen25 -j --target gtok   # builds libgtok.so
GBPE_VLLM=1 GBPE_GTOK_LIB=$PWD/build/qwen25/libgtok.so vllm serve Qwen/Qwen3-32B

Supported tokenizers

Each vocabulary has a CMake preset; the all preset and pip install . build GPT-2, Llama 3, DeepSeek-V3 and Gemma 3.

GPT-2 gpt2 Llama 3 llama3 Qwen 2.5 / Qwen 3 qwen25 DeepSeek-V3 deepseek_v3 Gemma 3 gemma3

Paper and citation

GraphTok is described in Taking Tokenization off the Host for Agentic LLM Serving (2026). A machine-readable CITATION.cff is in the repository.

arXiv link coming soon

@misc{siavashi2026graphtok,
  title={Taking Tokenization off the Host for Agentic {LLM} Serving},
  author={Siavashi, Mohammad and Schwinte, Pascal and Banaei, Ali and Maguire Jr., Gerald Q. and Chiesa, Marco and Kosti{\'c}, Dejan},
  year={2026},
  note={arXiv preprint}
}