Fix Perplexity Score For Tokenizers without bos_token_id (#682)
Update perplexity.py to handle models without bos_token_id in the tokeniser
K
Kyle Howells committed
f05792fff67a356f696fdf17ac64bd99602794c4
Parent: b39b266
Committed by GitHub <noreply@github.com>
on 6/20/2025, 4:02:27 PM