* fix ci
* fix stubs
* nit
* exclude
* full fix
* update
* up
* revert
* workflow up
* thius?
* up
* add logs I suspect its just maturin missing
* marutin not installed but not needed
* update
* check style after running tests since I mess up the .pyi
* nit?
* something that is supposed to work but my env does not allow it, seems to be uv related
* ?
* up
* nits
* let' s try
* part of tthe update for pyo3 0.27
* more pyo3 fixes
* update
* does this help?
* help
* finally
* update stub accordingly
* export more of the submodules
* moooore
* add individual .pypi
* cleanup
* update pyo3 signatures and fix warning
* style
* update
* more updates
* sytle
* clippy happy
* does this help?
* fix
* fix
* ?
* what?
* add dwarwub case co
* up?
* update
* clippy and fmt
* this time it works
* remove offending one
* update
* remove shit
* remove more shit that was unwanted
* ?
* simplify a bit
* more verbose?
* more simplification
* fmt
* fix some of the typing in rust directly to please TY (but also just fix some typing.Any
* fix script running
* fix , ignore and exclude
* style
* update
* fmt + add it to style?
* cleanup
* Simplify stub.py docstring injection
- Replace complex modifications dict with simple insertions list
- Remove nested process_function_or_method function
- Use bottom-to-top line replacement for cleaner logic
- Remove unused importlib import
* isolate stub generation into separate tools/stub-gen crate
- Move stub_generation.rs to tools/stub-gen/ as standalone crate
- Remove stub-gen feature and pyo3-introspection from main crate
- Auto-detect PYTHONHOME for uv/venv environments
- Update Makefile and README with new instructions
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* update stub for typing
* up
* add ty type checker
* update stub
* up
* some update
* add owner to stub?
* update
* no print
* uptime funk
* mm
* wtf
* fix
* fix more
* some fixses are manual but come on
* up
* # type: ignore[import]
* reduce the scope of ty for less changes
* ups
* up?
* Add benchmark for deserializing large added vocab
* revert dumb stuff, isolate changes
* try to only normalize once
* small improvement?
* some updates
* nit
* fmt
* normalized string are a fucking waste of time when you just want to add tokens to the vocab man....
* more attempts
* works
* let's fucking go, parity
* update
* hahahhahaha
* revert changes that are not actually even needed
* add a python test!
* use normalizer before come on
* nit
* update to a more concrete usecase
* fix build
* style
* reduce sample size
* --allow unmaintained
* clippy happy
* up
* up
* derive impl
* revert unrelated
* fmt
* ignore
* remove stupid file
* Testing ABI3 wheels to reduce number of wheels
* No need for py-clone anymore.
* Upgrade python versions.
* Remove those flakes.
* Promoting new CI + Fixing secret.
* Update docs-check.yml
Bump actions/setup-python to v5
Bump python-version to 3.12 (default on ubuntu-latest)
Switch actions-rs/toolchain to dtolnay/rust-toolchain as the former one is no longer maintained
* Update node-release.yml
Bump actions/setup-python to v5
Switch actions-rs/toolchain to dtolnay/rust-toolchain as the former one is no longer maintained
Bump actions/cache to v4
Bump actions/setup-node to v4
Bump actions/upload-artifact to v4
Bump actions/download-artifact to v4
* Update node.yml
Switch actions-rs/toolchain to dtolnay/rust-toolchain as the former one is no longer maintained
Bump actions/cache to v4
Bump actions/setup-node to v4
* Update python-release-conda.yml
Switch actions-rs/toolchain to dtolnay/rust-toolchain as the former one is no longer maintained
Bump conda-incubator/setup-miniconda to v3
* Update python-release.yml
Bump actions/setup-python to v5
Bump actions/download-artifact to v4
* Update rust-release.yml
Switch actions-rs/toolchain to dtolnay/rust-toolchain as the former one is no longer maintained
Bump actions/cache to v4
* Update stale.yml
Bump actions/stale to v9
* Update python.yml
Bump actions/setup-python to v5
* [BREAKING CHANGE] Ignore added_tokens (both special and not) in the
decoder
Causes issues with `ByteLevel` messing up some `AddedTokens` with some
utf-8 range used in the bytelevel mapping.
This commit tests the extend of the damage of ignoring the decoder for
those tokens.
* Format.
* Installing cargo audit.
* Minor fix.
* Fixing "bug" in node/python.
* Autoformat.
* Clippy.
* Only prefix space when there's no decoder.
* remove enforcement of non special when adding tokens
* mut no longer needed
* add a small test
* nit
* style
* audit
* ignore cargo audit's own vulnerability
* update
* revert
* remove CVE
* Fixing the progressbar.
* Upgrade deps.
* Update cargo audit
* Ssh this action.
* Fixing esaxx by using slower rust version.
* Trying the new esaxx version.
* Publish.
* Get cache again.
* Move to maturing mimicking move for `safetensors`.
* Tmp.
* Fix sdist.
* Wat?
* Clippy 1.72
* Remove if.
* Conda sed.
* Fix doc check workflow.
* Moving to maturin AND removing http + openssl mess (smoothing transition
moving to `huggingface_hub`)
* Fix dep
* Black.
* New node bindings.
* Fix docs + node cache ?
* Yarn.
* Working dir.
* Extension module.
* Put back interpreter.
* Remove cache.
* New attempt
* Multi python.
* Remove FromPretrained.
* Remove traces of `fromPretrained`.
* Drop 3.12 for windows?
* Typo.
* Put back the default feature for ignoring links during simple test.
* Fix ?
* x86_64 -> x64.
* Remove warning for windows bindings.
* Excluse aarch.
* Include/exclude.
* Put back workflows in correct states.
* Include license file in Rust crate
* Ignore security warning.
* Also for python.
* Upgrading ubuntu version.
Co-authored-by: Nicolas Patry <patry.nicolas@protonmail.com>
* Adding rust audit.
* Update clap version + derive_builder (they clashed).
* Ignoring specific CVE which can be ignored
https://github.com/Azure/iot-identity-service/issues/481
* Updating python lock.
* Revert `derive-builder` update.
* Adding back help msg.
* Upgrade pyo3 to 0.15
Rebase-conflicts-fixed-by: H. Vetinari <h.vetinari@gmx.com>
* Upgrade pyo3 to 0.16
Rebase-conflicts-fixed-by: H. Vetinari <h.vetinari@gmx.com>
* Install Python before running cargo clippy
* Fix clippy warnings
* Use `PyArray_Check` instead of downcasting to `PyArray1<u8>`
* Enable `auto-initialize` of pyo3 to fix `cargo test
--no-default-features`
* Fix some test cases
Why do they change?
* Refactor and add SAFETY comments to `PyArrayUnicode`
Replace deprecated `PyUnicode_FromUnicode` with `PyUnicode_FromKindAndData`
Co-authored-by: messense <messense@icloud.com>
* First pass on automatic stubbing our python files.
* And now modifying all rust docs to be visible in Pyi files.
* Better assert fail message.
* Fixing github workflow.
* Removing types not exported anymore.
* Fixing `Tokenizer` signature.
* Disabling auto __init__.py.
* Re-enabling some types.
* Don't overwrite non automated __init__.py
* Automated most __init__.py
* Restubbing after rebase.
* Fixing env for tests.
* Install blakc in the env.
* Use PY35 target in stub.py
Co-authored-by: Anthony MOI <m.anthony.moi@gmail.com>
* testable usage docs for training and serialization and reference in README.md
* Generate Readme from testable examples + template
* add up-to-date check for Readme with generated one
* try make pipeline fail by adding something to the lib.rs readme
* remove difference from lib.rs again to make pipeline pass
* fix black version
Co-authored-by: Simon Ertl <simon@Simons-MacBook-Pro.local>
This allows testing versions not built in-place. Otherwise
importing (or testing) in the package root fails without develop
builds.
Replace maturin with setuptools_rust since maturin fails with
proper project structure.
