Repository navigation
About the bos token in Gemma4 finetune notebook #251
Description
In the notebook for Gemma4 (31B) Conversational, the leading <bos> token is removed using removeprefix('<bos>') when formatting the prompt, and is said that it will be added back before training.
However, while verifying whether the <bos> token is appeared only once in the training sample via tokenizer.decode(trainer.train_dataset[100]["input_ids"]), the result clearly shows no <bos> at all. In addition, I've also check that the result of tokenizer.tokenizer.add_bos_token is False.
Does this means that the <bos> token is truly disappeared during the training process?
Do we need to add it back to ensure that the prompt format is correct as expected?
Metadata
Metadata
Assignees
Labels
No labels
Activity
Ki-Seki commentedon Apr 24, 2026
Hi. I’ve also encountered this issue. Did you manage to resolve it? I suspect the Unsloth team might have made a mistake here.
We cannot see the bos in the printed training data example
Training loss indicates missing
<bos>cause model training collapseTraining Loss of GPT (loss starting from about 2~)
Training Loss of Qwen (loss starting from about 1~)
Training Loss of Gemma4 (loss starting from about 12~ !!!!🤯)
This is also the same in my experiments
Ki-Seki commentedon Apr 24, 2026
@danielhanchen
danielhanchen commentedon Oct 22, 2026
Fixed! Sorry on the delay. The Gemma 4 Text notebooks stripped
<bos>from the training text assuming the tokenizer would add it back, but the Gemma 4 tokenizer never does, so every training row started without<bos>while inference always starts with it. The notebooks now keep the<bos>the chat template adds, so each row has exactly one, fixed in #383. The very high starting loss on 26B / 31B was a separate issue (the missing empty thought channel on model turns), fixed in unslothai/unsloth#12867.To update:
uv pip install --upgrade unsloth unsloth_zoo --force-reinstall --no-cache-dirand open the notebook again from the repo (if it's not in the current release yet it'll be in the next one). Reopen if it still happens.Gemma 4 Text notebooks: keep <bos> in the training text (unslothai#383)