💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
COMMITS
/ README.md August 29, 2025
A
Fix typo in README (#1808)
AN Long committed
November 15, 2024
A
Update README.md with install from source
Arthur committed
August 12, 2024
1
fix filelink (#1610)
152334H committed
August 9, 2024
A
Update README.md (#1608)
Arthur committed
December 9, 2023
P
Fix doc links in readme (#1367)
Pierric Cistac committed
July 12, 2023
A
master -> main (#1292)
Arthit Suriyawongkul committed
June 8, 2023
S
Update README.md - Broken link (#1272)
Santosh Bhavani committed
March 24, 2022
N
Adding a link to the Ruby port of `tokenizers` (#961)
Nicolas Patry committed
February 10, 2021
A
Actually fix the link to pepy.tech on downloads badge
Anthony MOI committed
A
Hotfix link to pepy.tech on downloads badge
Anthony MOI committed
February 3, 2021
A
Update README.md
Anthony MOI committed
January 6, 2021
J
rm extraneous </a> (#573)
Julien Chaumond committed
November 4, 2020
N
Fixing the readme link
Nicolas Patry committed
November 2, 2020
A
Doc - Add @narsil suggestions
Anthony MOI committed
N
Update README.md
Nicolas Patry committed
October 8, 2020
S
Update README
Sylvain Gugger committed
September 25, 2020
N
Attempt to get some documentation going.
Nicolas Patry committed
February 18, 2020
A
Python - Replace last BPETokenizer occurences
Anthony MOI committed
February 11, 2020
0
fixed corresponding typo
0xflotus committed
February 7, 2020
C
Move up the binding in the readme
Clement committed
C
Add contributors-hall-of-fame
Clement committed
January 14, 2020
A
Typo in README.md
Alan deLevie committed
January 13, 2020
A
Update README.md
Anthony MOI committed
B
Fix a few errors in the README.md
benjamin-ny committed
January 10, 2020
C
fix link counter
Clement committed
C
Adding download counter
Clement committed
J
Update README.md
Julien Chaumond committed
M
Update README
MOI Anthony committed
M
Readme - remove bad link
MOI Anthony committed
M
Add rust to readme
MOI Anthony committed
M
Tweak readme
MOI Anthony committed
M
Improve readme
MOI Anthony committed
A
Split readme
Anthony MOI committed
January 8, 2020
A
Quick README update
Anthony MOI committed
January 7, 2020
E
small wording changes (#39)
Evan Pete Walsh committed
January 5, 2020
A
Documentation updates
Anthony MOI committed
November 18, 2019
M
Fix readme indentation
MOI Anthony committed
A
Update readme with simple example
Anthony MOI committed
November 1, 2019
A
Rust install
Anthony MOI committed
A
Initial commit
Anthony MOI committed