💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
COMMITS
/ bindings/python/Cargo.lock November 4, 2024
A
cargo lock
Arthur Zucker committed
December 25, 2022
R
Ignore Cargo.lock for subfolders (#1131)
Roy Hvaara committed
November 9, 2022
N
Adding rust audit. (#1099)
Nicolas Patry committed
November 7, 2022
C
Add python 3.11 to manylinux buildwheels (#1096)
Cameron committed
October 5, 2022
D
pyo3: update to 0.17 (#1066)
David Hewitt committed
September 26, 2022
N
Fixing roberta type id (everything is zero). (#1072)
Nicolas Patry committed
September 21, 2022
N
Enabling static interpreter embedding for manylinux. (#1064)
Nicolas Patry committed
September 12, 2022
N
Preparing rc1 release. (#1056)
Nicolas Patry committed
August 24, 2022
N
Updating code according to clippy. (#1048)
Nicolas Patry committed
June 10, 2022
N
Adding `unstable_wasm` feature + example to run `tokenizers` on wasm. (#1009)
Nicolas Patry committed
June 2, 2022
N
Changing `Decoder` trait to be more composable. (#938) (#1008)
Nicolas Patry committed
May 5, 2022
H
Upgrade pyo3 to 0.16 (#956)
h-vetinari committed
March 21, 2022
N
Fixing issue with ConvBert not being able to save because of of holes in (#954)
Nicolas Patry committed
February 28, 2022
N
Workarounds publishing issues:
Nicolas Patry committed
February 15, 2022
T
tokenizer.save has the wrong arguments compared to documentation (#901)
Thomas Wang committed
January 17, 2022
N
Update Cargo.lock for Python.
Nicolas Patry committed
January 6, 2022
N
Aho corasick version for many added tokens. (#871)
Nicolas Patry committed
September 8, 2021
A
Python - Prepare for release 0.11.0 (#799)
Anthony MOI committed
September 2, 2021
A
Python - Update Cargo.lock
Anthony Moi committed
August 19, 2021
A
Python - Add bindings to Tokenizer.from_pretrained
Anthony Moi committed
August 13, 2021
G
Switch git dependencies in Cargo.toml back to regular versions (#728)
Geoffrey Thomas committed
July 21, 2021
N
Fixing tokenizers with 1.53 (updated some dependencies + clippy) (#764)
Nicolas Patry committed
May 24, 2021
A
Python - prepare for release 0.10.3
Anthony MOI committed
April 5, 2021
A
Python - Prepare for release 0.10.2
Anthony MOI committed
February 8, 2021
A
Prepare for python v0.10.1 (#625)
Anthony MOI committed
February 4, 2021
A
Python - Prepare for release 0.10.1rc1 (#622)
Anthony MOI committed
January 12, 2021
A
Python - Prepare for release 0.10.0
Anthony MOI committed
December 15, 2020
A
Python - Improve training with iterators
Anthony MOI committed
December 4, 2020
A
Python - Update CHANGELOG and bump for release
Anthony MOI committed
November 25, 2020
A
Also accept iterators of batches in train_from_iterator
Anthony MOI committed
November 12, 2020
A
Add ability to train from Iterator
Anthony MOI committed
November 11, 2020
N
New PR to fix #270 (not #157). (#516)
Nicolas Patry committed
November 9, 2020
A
Python - Update CHANGELOG and bump version for 0.9.4
Anthony MOI committed
October 26, 2020
A
Python - Update CHANGELOG and bump to 0.9.3 for release
Anthony MOI committed
October 15, 2020
A
Python - Update CHANGELOG and bump to 0.9.2 for release
Anthony MOI committed
October 13, 2020
A
Python - Update CHANGELOG and bump version for release
Anthony MOI committed
October 9, 2020
A
Python - Update dependencies for release
Anthony MOI committed
A
Python - Bump version for 0.9.0 release
Anthony MOI committed
September 29, 2020
A
Python - Update CHANGELOG and bump to 0.9.0.rc1
Anthony MOI committed
September 24, 2020
A
Python - Bump version for dev4 release
Anthony MOI committed
September 18, 2020
A
Python - Add bindings for NormalizedString
Anthony MOI committed
N
Move to spm_precompiled 0.1.2 for smaller binary string.
Nicolas Patry committed
September 22, 2020
A
Update PyO3 (#426)
Anthony MOI committed
September 17, 2020
N
Fixed vietnamese bug, now we have a thai bug.
Nicolas Patry committed
September 15, 2020
N
Improvements on spm parity: (#401)
Nicolas Patry committed
September 14, 2020
N
Upgrading dependencies (esaxx-rs to build).
Nicolas Patry committed
September 3, 2020
N
Adding a new pre_tokenizer: Digits.
Nicolas Patry committed
August 21, 2020
N
Failing test for compatibility for SentencePieceUnigramTokenizer.
Nicolas Patry committed
August 31, 2020
A
Add failing test for from_file
Anthony MOI committed