vocab : honor BOS/EOS settings for PLaMo-2 and PLaMo-3 (#29734)
* vocab : honor BOS/EOS settings for PLaMo-2 and PLaMo-3 The original tokenizer configs for PLaMo-2 and PLaMo-3 have `add_bos_token: true` and `add_eos_token: false` , but _set_vocab_plamo() did not write the BOS/EOS metadata. The PLAMO2 tokenizer path also ignored add_bos/add_eos during tokenization. Write the settings from tokenizer_config.json and honor them in the PLAMO2 tokenization path. GGUFs without these keys keep the previous behavior. * Update conversion/base.py Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co> --------- Co-authored-by: Sigbjørn Skjæret <sigbjorn.skjaeret@huggingface.co>
T
Toki Nasin committed
db33d3cb89d8b0d954df66047b13e9cc22df8f10
Parent: 7dad6db
Committed by GitHub <noreply@github.com>
on 10/1/2026, 5:31:42 AM