Commit Graph

Select branches

Hide Pull Requests

add-added-tokens-encoder

add-some-doc

add_decoders_to_doc

assign-token

austinleedavis/main

backlog_release

backtrack

bench-python

big_science_fork2

bigscience_fork

branch_0.13.4.rc2

branch_0.14.0

branch_0.15.0

branch_0.19.1

branch_0.20.1.rc1

branch_v0.14.0.rc1

branch_v0.15.1.rc1

branch_v0.15.2.rc1

branch_v0.16.0.rc1

branch_v0.20.0.rc1

cache-evmap

change_decoder_trait

ci-test-huggingface-hub-v1.0.0.rc0

clefourrier-patch-version-dep-hub

clippy_update

copilot/remove-stderr-warning-python-bindings

debug-release-ci

disable-par-iter

doc_build_typo

docs_new_frontend

extra-tokens

extra_wheel

fast-encode

fast-encode-2

fast-regex

feat/early_exit_right_truncation

feat/improve_whitespace_splitting_perf

feat/truncate_overflow_with_stride_test

feature/post-process-tokens

feature/role-to-token

feature/special-tokens-api

feature/unigram-unk-token

fix-cache-issues

fix-merge-words

fix-pretokenizer

fix-split-special

fix_ahash_pub

fix_anaconda

fix_benchmark2

fix_manylinux

fix_special_tokens_mask

fixing_decoder_bug_before_release

git_0.14.1

git_0.14.1rc1

git_0.15.1rc0

git_0.19.0

git_0.19.0rc0

git_v0.20.3-rc0

git_v0.20.4-rc0

git_v0.21.0

git_v0.21.1

git_v0.21.1_rc0

git_v0.21.3

git_v0.21.4

git_v0.22.0

impl_serde_type_struct

java-binding

main

metaspace_nosplit

mishig25-patch-1

mishig25-patch-2

model-config

modify_processor_trait

narsil/backtrack

patched

pep440

pickling_trainers

pin_docutils

prepare_patch

python-313-release

refactor_node

remove_unecessary_yelling

revert-1047-modify_processor_trait

serde_pyo3

skip_tokens_too_long_bpe

template_post_processor

test-old-decode

thomas/get_white_space_regex_with_byte_level

thomas/no_regex_byte_level

thomas/unigram_parallel_e_step

tmp

tmp_release

tmpbytefallback_hack

tryout

u64_branch

unigram_bug_maybe

update-serialize

update-stream

update-stuff

update_rand_and_clap_rebase

updating_replace_normalized_string_behavior

used_vendored_for_manylinux

v0.13.0-rc1

v0.20.3-release

v0.21.2

v0.21.2_rc0

v0.22.0-rc0

v0.22.1-rc0

v0.22.2-rc0

windows_aarch

wordpiece_aho

#1

#1005

#1008

#1009

#101

#1019

#102

#1021

#1024

#103

#1030

#1032

#1034

#1038

#1047

#1048

#1049

#1051

#1052

#1053

#1054

#1055

#1056

#1060

#1063

#1064

#1066

#1067

#1069

#107

#1072

#1079

#1081

#1082

#109

#1095

#1096

#1098

#1099

#110

#1101

#1102

#1107

#1108

#111

#1115

#1116

#112

#1120

#1123

#1124

#1125

#1126

#1127

#1129

#1130

#1131

#1133

#1134

#1136

#1137

#1139

#114

#1140

#1142

#1144

#1145

#1147

#1148

#1150

#1152

#1153

#1154

#1157

#116

#1166

#1167

#117

#1173

#118

#1181

#1182

#1183

#1188

#1189

#1191

#1192

#1194

#1195

#1196

#1197

#1199

#1203

#1205

#1207

#1209

#1209

#121

#1210

#1211

#1212

#1217

#1222

#1227

#1228

#1233

#1235

#124

#1244

#1245

#1246

#1248

#125

#1251

#1254

#1255

#1255

#1256

#1257

#126

#1262

#1264

#1266

#1268

#1270

#1271

#1272

#1273

#1275

#128

#1288

#1289

#129

#1291

#1292

#1293

#1294

#1295

#1296

#1299

#1302

#1303

#1306

#1308

#131

#1312

#1316

#1317

#1318

#1319

#132

#1320

#1321

#1322

#1328

#1329

#1330

#1331

#1333

#1335

#1338

#1339

#134

#1341

#1344

#1353

#1355

#1356

#1357

#136

#1367

#1370

#1371

#1377

#1378

#1379

#138

#1381

#1382

#1383

#1385

#1386

#1388

#1390

#1394

#1396

#140

#1404

#1406

#1413

#1414

#1419

#1420

#1428

#1430

#1433

#1435

#1436

#1437

#1438

#1441

#1442

#1443

#1444

#1453

#1456

#1459

#146

#1462

#1465

#147

#1472

#1476

#1478

#1479

#1482

#1483

#1484

#1486

#1488

#149

#1492

#1493

#1494

#1497

#1498

#1499

#1503

#1504

#1508

#1510

#1511

#1513

#1521

#1532

#1533

#1535

#1540

#1542

#1547

#1548

#155

#1550

#1551

#1554

#1555

#1557

#1558

#1560

#1561

#1568

#1569

#1570

#1570

#1571

#1574

#1580

#1582

#1583

#1584

#1585

#1586

#1587

#1588

#1589

#1590

#1591

#1592

#1593

#1594

#1595

#1597

#1598

#1599

#1600

#1602

#1603

#1604

#1605

#1605

#1606

#1607

#1608

#1610

#1617

#1618

#162

#1621

#1622

#1626

#1629

#1630

#1631

#1632

#1638

#1640

#1641

#1642

#1646

#1647

#1648

#165

#1651

#1652

#1653

#1659

#1660

#1662

#1664

#1665

#1667

#1669

#167

#1670

#1673

#1674

#1675

#1676

#1677

#1678

#1679

#1685

#1687

#1689

#169

#1693

#1699

#1701

#1707

#1708

#1709

#171

#1710

#1711

#1712

#1712

#1713

#1714

#1715

#1716

#1717

#1719

#1719

#172

#1724

#1725

#1732

#1733

#1733

#1737

#1739

#1744

#1745

#1746

#1747

#1750

#1752

#1753

#1753

#1755

#1755

#1756

#1758

#1762

#1763

#1763

#1764

#1766

#1770

#1771

#1772

#1774

#1778

#1780

#1781

#1782

#1783

#1783

#1785

#1788

#1788

#1789

#1789

#179

#1792

#1796

#1798

#1799

#18

#180

#1800

#1801

#1802

#1804

#1804

#1805

#1806

#1808

#1809

#1814

#1818

#1822

#1822

#1827

#183

#1830

#1830

#1831

#1832

#1832

#1834

#1838

#184

#1840

#1841

#1841

#1842

#1842

#1843

#1844

#1846

#1848

#1849

#1852

#1853

#1856

#1857

#1859

#1859

#1860

#1861

#1862

#1862

#1864

#1865

#1866

#1867

#1869

#1872

#1874

#1875

#1878

#188

#1880

#1880

#1883

#1884

#1888

#1888

#1889

#1891

#1892

#1893

#1893

#1894

#1894

#1895

#1895

#1896

#1898

#1899

#19

#190

#1901

#1903

#1905

#1907

#1911

#1912

#1912

#1916

#1916

#1921

#1923

#1924

#1925

#1928

#193

#1930

#1931

#1931

#1932

#1932

#1934

#1935

#1936

#1937

#1938

#1939

#1940

#1941

#1942

#1942

#1943

#1943

#1944

#1944

#1945

#1945

#1946

#1948

#1949

#1950

#1951

#1952

#1952

#1953

#1955

#1956

#1958

#1959

#1959

#196

#1963

#1964

#1965

#1965

#1966

#1967

#1967

#1968

#1968

#1969

#197

#1970

#1970

#1971

#1972

#1974

#1974

#1976

#1976

#1977

#1978

#1979

#1980

#1981

#1982

#1983

#1984

#1985

#1985

#1986

#1987

#1988

#1989

#1990

#1990

#2

#20

#200

#202

#206

#208

#21

#212

#213

#22

#220

#221

#222

#224

#226

#227

#229

#23

#234

#236

#238

#239

#24

#241

#249

#25

#26

#27

#271

#272

#273

#274

#276

#277

#28

#280

#286

#289

#29

#292

#298

#3

#303

#306

#308

#309

#311

#312

#315

#32

#327

#329

#33

#330

#333

#334

#338

#34

#344

#346

#347

#348

#350

#355

#356

#357

#36

#360

#362

#363

#364

#365

#366

#368

#37

#373

#377

#378

#379

#38

#383

#384

#385

#386

#387

#388

#389

#39

#390

#392

#393

#394

#395

#4

#40

#400

#401

#403

#407

#41

#416

#417

#42

#420

#421

#426

#427

#428

#429

#43

#430

#433

#434

#436

#44

#440

#441

#442

#445

#448

#45

#452

#453

#454

#455

#459

#462

#463

#464

#467

#47

#470

#473

#476

#477

#48

#480

#481

#485

#49

#490

#492

#494

#495

#498

#5

#50

#500

#503

#506

#508

#509

#512

#513

#514

#516

#518

#519

#520

#530

#533

#538

#539

#542

#543

#544

#549

#55

#551

#557

#559

#561

#565

#569

#573

#574

#582

#583

#589

#590

#591

#593

#594

#595

#596

#6

#60

#610

#616

#617

#618

#620

#621

#622

#625

#64

#650

#652

#653

#656

#657

#660

#671

#674

#675

#676

#68

#686

#691

#693

#7

#70

#700

#701

#702

#707

#708

#71

#713

#714

#716

#718

#719

#720

#721

#727

#728

#734

#743

#75

#758

#76

#762

#763

#764

#768

#77

#770

#771

#773

#774

#775

#78

#780

#789

#792

#793

#794

#795

#799

#8

#80

#806

#825

#831

#832

#833

#837

#841

#842

#844

#845

#846

#850

#851

#856

#857

#858

#859

#860

#863

#864

#868

#869

#870

#871

#877

#881

#882

#884

#887

#890

#894

#895

#896

#90

#901

#904

#907

#908

#909

#913

#915

#916

#919

#92

#920

#921

#923

#924

#925

#927

#93

#936

#937

#938

#939

#94

#940

#942

#943

#949

#95

#951

#952

#954

#956

#958

#960

#961

#962

#963

#966

#967

#969

#971

#973

#975

#976

#978

#980

#982

#986

#987

#988

#989

#99

#992

node-v0.12.0

node-v0.13.0

node-v0.13.1

node-v0.13.2

node-v0.13.3

node-v0.2.3

node-v0.2.4

node-v0.3.0

node-v0.3.1

node-v0.4.0

node-v0.4.1

node-v0.5.0

node-v0.6.0

node-v0.6.1

node-v0.6.2

node-v0.7.0

node-v0.8.0

node-v0.8.1

node-v0.8.2

node-v0.8.3

python-v0.10.0

python-v0.10.0rc1

python-v0.10.1

python-v0.10.1rc1

python-v0.10.2

python-v0.10.3

python-v0.11.0

python-v0.11.1

python-v0.11.2

python-v0.11.3

python-v0.11.4

python-v0.11.5

python-v0.11.6

python-v0.12.0

python-v0.12.1

python-v0.13.0

python-v0.13.1

python-v0.13.2

python-v0.13.3

python-v0.13.3rc1

python-v0.2.0

python-v0.2.1

python-v0.3.0

python-v0.4.0

python-v0.4.1

python-v0.4.2

python-v0.5.0

python-v0.5.1

python-v0.5.2

python-v0.6.0

python-v0.7.0

python-v0.7.0-rc1

python-v0.7.0-rc2

python-v0.7.0-rc3

python-v0.7.0-rc4

python-v0.7.0-rc5

python-v0.7.0-rc6

python-v0.7.0-rc7

python-v0.8.0

python-v0.8.0-dev

python-v0.8.0-dev1

python-v0.8.0-dev2

python-v0.8.0-rc1

python-v0.8.0-rc2

python-v0.8.0-rc3

python-v0.8.0-transformers

python-v0.8.1

python-v0.8.1.rc1

python-v0.8.1.rc2

python-v0.9.0

python-v0.9.0.dev0

python-v0.9.0.dev1

python-v0.9.0.dev2

python-v0.9.0.dev3

python-v0.9.0.dev4

python-v0.9.0.rc1

python-v0.9.0.rc2

python-v0.9.1

python-v0.9.2

python-v0.9.3

python-v0.9.4

rust-v0.10.0

rust-v0.10.1

rust-v0.8.0

rust-v0.9.0

v0.0.10

v0.0.11

v0.0.12

v0.0.13

v0.0.3

v0.0.4

v0.0.5

v0.0.6

v0.0.7

v0.0.8

v0.0.9

v0.1.0

v0.1.1

v0.11.1

v0.11.2

v0.12.0

v0.13.0

v0.13.1

v0.13.2

v0.13.3

v0.13.4-rc1

v0.13.4-rc2

v0.13.4.rc1

v0.13.4.rc2

v0.13.4.rc3

v0.14.0

v0.14.0.rc1

v0.14.1

v0.14.1rc1

v0.15.0

v0.15.0.rc1

v0.15.1

v0.15.1.rc0

v0.15.1.rc1

v0.15.1rc0

v0.15.2

v0.15.2rc1

v0.16.0rc1

v0.19.0

v0.19.0rc0

v0.19.1

v0.19.1rc0

v0.20.0

v0.20.0rc1

v0.20.1

v0.20.1rc1

v0.20.2

v0.20.2rc1

v0.20.3

v0.20.3rc1

v0.20.4

v0.20.4-rc0

v0.20.4rc0

v0.21.0

v0.21.0rc0

v0.21.1

v0.21.1rc0

v0.21.2

v0.21.2rc0

v0.21.3

v0.21.4

v0.22.0

v0.22.0rc0

v0.22.1

v0.22.1rc0

v0.22.2

v0.22.2rc0

Mono Color

• 2208ba2a2f feat(tokenizer): add early exit for left truncation feat/early_exit_right_truncation Luc Georges 2026-03-26 19:45:57 +01:00
• a7fb4e2867 refactor(tokenizer): typo in tokenize_with_limit fn name Luc Georges 2026-03-26 19:14:44 +01:00
• ba3b7bcbde fix(tokenizer): skip early exit for OnlySecond strategy Luc Georges 2026-03-26 19:05:33 +01:00
• 45f431ee8c feat(tokenizer): add early exit for right truncation Luc Georges 2026-03-26 18:43:47 +01:00
• d1ac6e9654 Final doc fix (#1989) main Arthur 2026-03-26 17:51:09 +01:00
• 22823c7803 update (#1988) Arthur 2026-03-26 17:35:30 +01:00
• 88f4f79f36 Fix doc builds (#1987) Arthur 2026-03-26 17:25:05 +01:00
• bd19f4b997 Fix doc builds (#1986) Arthur 2026-03-26 16:56:50 +01:00
• a09f44bf3b refactor: fmt feat/improve_whitespace_splitting_perf Luc Georges 2026-03-26 16:48:19 +01:00
• 1ae7d0ed16 feat: add new faster whitespace split pretok Luc Georges 2026-03-26 16:26:46 +01:00
• 44ccca5abd Fix doc builds (#1984) Arthur 2026-03-26 16:15:26 +01:00
• a5a221069d Fix doc builds (#1983) Arthur 2026-03-26 16:06:08 +01:00
• d36499e3aa Fix doc builds (#1982) Arthur 2026-03-26 15:40:54 +01:00
• 229bdf3d7a add async decode batch (#1966) Michael Feil 2026-03-26 04:07:32 -07:00
• 67595efe57 Bump picomatch in /tokenizers/examples/unstable_wasm/www (#1979) dependabot[bot] 2026-03-26 12:00:38 +01:00
• bd3df8bf45 fix binding test post merge (#1981) Arthur 2026-03-26 11:55:52 +01:00
• b432bc3769 let' s check something tryout Arthur 2026-03-26 11:26:58 +01:00
• ec3e249115 Mark Python tests that need network access (#1872) Gordon Messmer 2026-03-26 06:19:30 -04:00
• 9bbf770020 feat: add progress_format option for machine-readable JSON output (#1921) Andrii Podanenko 2026-03-26 12:02:46 +02:00
• 44a84169fd Add riscv64 build, make Linux wheel build matrix more explicit (#1951) Trevor Gamblin 2026-03-26 05:46:45 -04:00
• 36ea27b863 Bump minimatch from 3.1.3 to 3.1.5 in /bindings/node (#1956) dependabot[bot] 2026-03-26 08:08:31 +01:00
• 7c4006413d Bump flatted from 3.3.1 to 3.4.2 in /bindings/node (#1972) dependabot[bot] 2026-03-26 08:07:25 +01:00
• cd5c8f09e1 Bump picomatch from 2.3.1 to 2.3.2 in /bindings/node (#1980) dependabot[bot] 2026-03-26 08:07:04 +01:00
• fbf1f1aed8 feat: Add weakref support to Tokenizer class (#1958) Shintaro Murakami 2026-03-26 06:51:37 +09:00
• e0502118e6 Fix broken source links in documentation (#1934) Shivam 2026-03-25 10:22:53 -07:00
• 2044dcac0b Fix identity comparison in toctree_tags.py (#1949) Luka Aladashvili 2026-03-25 21:21:34 +04:00
• d95b505010 feat(benchmarks-only): adding longer-context llama3 benchmarks (#1971) Michael Feil 2026-03-25 10:13:55 -07:00
• 8ec1976d19 fix ci (#1978) Arthur 2026-03-25 18:13:04 +01:00
• 9d89977060 feat: add support for uv in Makefile (#1977) Luc Georges 2026-03-25 15:41:12 +01:00
• cbd8cf251e Remove unnecessary to_vec() from slice() (#1964) J Berg 2026-03-24 14:15:21 +00:00
• c4e27cfc84 Fix multithreaded concurrency test to use shared tokenizer instance (#1950) Shintaro Murakami 2026-02-28 01:54:02 +09:00
• c370063d8e Bump minimatch from 3.1.2 to 3.1.3 in /bindings/node (#1955) dependabot[bot] 2026-02-25 12:04:21 +01:00
• 4c2e48ab2f Update to PyO3 0.28 to automatically disable GIL (#1948) Nathan Goldbaum 2026-02-25 02:12:20 -07:00
• c58a47475c docs: add all features (#1953) Wayne Lau 2026-02-24 17:55:05 +08:00
• fbff6cab7e revert feature/role-to-token Arthur 2026-02-20 11:21:26 +01:00
• 6fbda4d6a9 just make it happy for now Arthur 2026-02-19 11:20:49 +01:00
• c01b781396 skip musli arm7 for now Arthur 2026-02-19 11:18:25 +01:00
• e535285039 more fmt Arthur 2026-02-19 11:17:27 +01:00
• 631af6f5af fmt bindings Arthur 2026-02-19 11:17:12 +01:00
• 3a6ba371dc fix adding a token as not special when it was special was still special Arthur 2026-02-19 11:10:05 +01:00
• 8f34aaf653 test more Arthur 2026-02-19 10:47:33 +01:00
• b1aa45cf8e update Arthur 2026-02-19 10:40:28 +01:00
• bf41f0f22d try to update auto with maturin Arthur 2026-02-19 09:23:27 +01:00
• 54d16ef9c6 clippy happy Arthur 2026-02-19 08:58:20 +01:00
• c1a2e2974f fix test Arthur 2026-02-18 17:30:54 +01:00
• 94ce5da7c7 fmt Arthur 2026-02-18 17:29:45 +01:00
• 01dd6f246e fmt Arthur 2026-02-18 15:26:10 +01:00
• bf6d45ab7c settatr Arthur 2026-02-18 15:26:00 +01:00
• a0b109ac5f add setattr Arthur 2026-02-18 15:24:32 +01:00
• 2b6b91154a move func + add support for _id Arthur 2026-02-18 15:08:54 +01:00
• f56aa2fe52 will this properly expose role-to-token? Arthur 2026-02-18 14:59:49 +01:00
• 511224ee1b remove somet stuff Arthur 2026-02-18 14:26:32 +01:00
• 4ee98861bb Merge branch 'main' of github.com:huggingface/tokenizers into feature/role-to-token Arthur 2026-02-18 14:15:39 +01:00
• 50352f73a5 Add type hint, update to pyo3 0.27, add automatic type hint generator (#1928) Arthur 2026-02-11 14:26:43 +01:00
• 6eba494a37 Add a multithreaded tokenizer test and as well as 3.14t CI (#1864) Nathan Goldbaum 2026-02-11 03:42:28 -07:00
• d3b76e2e5d Add windows arm64 wheel build to python release (#1907) Finn Womack 2026-02-11 02:30:03 -08:00
• 206ebb588f Fix escape HTML characters in EncodingVisualizer output (#1937) 大橋 玲音 2026-02-10 02:08:43 +09:00
• ec119e3f31 revertt and simplify just QOL feature/unigram-unk-token Arthur 2026-02-06 11:06:36 +01:00
• 9c8b066dcb Bump webpack in /tokenizers/examples/unstable_wasm/www (#1946) dependabot[bot] 2026-02-06 09:45:30 +01:00
• 33917e999e fix: use IPython.display instead of deprecated IPython.core.display (#1936) 大橋 玲音 2026-02-06 17:22:22 +09:00
• 995a477f25 Add Python 3.14 CI (#1925) Nathan Goldbaum 2026-02-06 01:21:02 -07:00
• 0ef35f1f6d Update fancy-regex dependency to 0.17 (#1940) Ben Beasley 2026-02-06 08:09:19 +00:00
• 2fa762e93e Add get_special_tokens and is_special_token methods feature/special-tokens-api Arthur 2026-02-05 15:02:42 +01:00
• 9988ba5a7b Add post_process_tokens and post_process_ids methods feature/post-process-tokens Arthur 2026-02-05 14:50:32 +01:00
• 31ef73cfca feat: add unk_token property to Unigram model Arthur 2026-02-05 09:27:49 +01:00
• 7c781c9975 feat: add role_to_token field for special token metadata Arthur 2026-02-05 09:01:41 +01:00
• 9c57a8505e Add missing test dependency (#1938) Alexander Lent 2026-02-02 04:57:57 -05:00
• 7938d8c857 Bump lodash from 4.17.21 to 4.17.23 in /bindings/node (#1935) dependabot[bot] 2026-01-28 15:48:27 +01:00
• b3889ab98a Add copy to a decostream (#1930) Arthur 2026-01-19 11:07:08 +01:00
• b874abe731 Fix warnings: remove a print and remove some deprecation warnings (#1924) Arthur 2026-01-05 14:02:44 +01:00
• a2fe1cc0a9 use macos-15-intel (#1923) Arthur 2026-01-05 13:28:34 +01:00
• f383101a26 fix max build? v0.22.2 v0.22.2-rc0 Arthur 2026-01-05 11:33:42 +01:00
• fb691515d0 Python release fix (#1905) Finn Womack 2026-01-05 02:09:46 -08:00
• e60793ef1c Python release fix (#1905) Finn Womack 2026-01-05 02:09:46 -08:00
• ecad3f18a3 Fix unclosed annotation span in EncodingVisualizer (#1911) 大橋 玲音 2025-12-16 19:52:44 +09:00
• f7db48f532 dev new version Arthur 2025-12-02 14:02:13 +01:00
• 6573f2c561 add lock Arthur 2025-12-02 13:52:55 +01:00
• 1d9dbb82a3 push the release Arthur 2025-12-02 13:51:12 +01:00
• 95504c0293 add .lock and v0.22.2 v0.22.2rc0 Arthur 2025-12-02 13:36:57 +01:00
• 8604740782 update stub for typing (#1896) Arthur 2025-12-02 12:48:56 +01:00
• a5e03bab57 Bump express in /tokenizers/examples/unstable_wasm/www (#1903) dependabot[bot] 2025-12-02 12:10:57 +01:00
• ebbc3c8da3 bump PyO3 to 0.26 (#1901) David Hewitt 2025-12-02 10:55:55 +00:00
• b83d7c986c DOCS: add add_prefix_space to processors.ByteLevel (#1878) Tobias Pitters 2025-11-28 18:38:49 +01:00
• 060786018e Mark immutable pyclasses as frozen (#1861) Nathan Goldbaum 2025-11-28 08:41:25 -07:00
• 09dafe2f44 Remove runtime stderr warning from Python bindings (#1898) Copilot 2025-11-28 13:52:00 +01:00
• 89d271becc Initial plan copilot/remove-stderr-warning-python-bindings copilot-swe-agent[bot] 2025-11-28 11:07:57 +00:00
• c47328ce54 fmt austinleedavis/main Arthur 2025-11-28 08:53:07 +01:00
• 23a5c16f0c Merge branch 'main' into main Arthur 2025-11-28 08:27:59 +01:00
• a05b60c55f [MINOR:TYPO] Update mod.rs (#1883) Christopher Akiki 2025-11-28 08:01:52 +01:00
• bc6e6cd09a fix: used normalize_str in BaseTokenizer.normalize (#1884) Ishita Bhattacharyya 2025-11-28 12:21:28 +05:30
• d6a4acc0d2 Update serialization (#1891) Arthur 2025-11-27 23:07:18 +01:00
• 47e4ffebab Bump js-yaml from 3.14.1 to 3.14.2 in /bindings/node (#1892) dependabot[bot] 2025-11-27 22:33:44 +01:00
• 904bbd0afa Bump node-forge in /tokenizers/examples/unstable_wasm/www (#1889) dependabot[bot] 2025-11-27 19:32:58 +01:00
• 849155249b Update indicatif dependency (#1867) Gordon Messmer 2025-11-27 10:15:51 -08:00
• 007fc767ac Add cargo-semver-checks to Rust CI workflow (#1875) Haixuan Xavier Tao 2025-10-16 11:22:48 +02:00
• 386f3d8267 ci: add support for building Win-ARM64 wheels (#1869) MUGUNDAN 2025-10-16 14:19:18 +05:30
• 14e1e97d4c u32 extra-tokens Arthur 2025-10-08 10:23:00 +02:00
• ea0da14c71 current draft! Arthur 2025-10-07 16:19:41 +02:00
• 916df54268 update dev version to 0.22.2 Arthur 2025-09-19 11:46:04 +02:00
• afaae08883 push the minor v0.22.1 v0.22.1-rc0 Arthur 2025-09-19 11:44:39 +02:00