MORPH
SIGN IN SIGN UP
huggingface / tokenizers UNCLAIMED

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

0 0 0 Rust

BRANCHES

20 branches
main DEFAULT
911b9aa
austinleedavis/main
c47328c
ci-test-huggingface-hub-v1.0.0.rc0
d33803c
copilot/remove-stderr-warning-python-bindings
89d271b
extra-tokens
14e1e97
feat/early_exit_right_truncation
2208ba2
feat/improve_whitespace_splitting_perf
a09f44b
feature/post-process-tokens
9988ba5
feature/role-to-token
fbff6ca
feature/special-tokens-api
2fa762e
feature/unigram-unk-token
ec119e3
fix-byte-norm
1188bdc
git_v0.21.4
e892882
git_v0.22.0
4630f94
regex-split-parity
3989017
tryout
b432bc3
update-doc
c0a9bc4
v0.22.0-rc0
1d34d66
v0.22.1-rc0
afaae08
v0.22.2-rc0
f383101