bindings/python/src - huggingface/tokenizers - Morph

SIGN IN SIGN UP

huggingface / tokenizers UNCLAIMED

💥 Fast State-of-the-Art Tokenizers optimized for Research and Production

0 0 0 Rust

	utils
	decoders.rs	29.7 KB
	encoding.rs	17.8 KB
	error.rs	1.2 KB
	lib.rs	2.9 KB
	models.rs	33.7 KB
	normalizers.rs	36.2 KB
	pre_tokenizers.rs	40.9 KB
	processors.rs	31.9 KB
	token.rs	1.1 KB
	tokenizer.rs	72.5 KB
	trainers.rs	36.1 KB