What special: true Actually Changes

I found the question in my own eval script, which is the embarrassing way to find things.

1
2

text = tokenizer.decode(output[0, input_len:], skip_special_tokens=True).strip()
text = re.sub(r"<think>.*?</think>\s*", "", text, flags=re.DOTALL).strip()

Decode with skip_special_tokens=True, then a hand-rolled regex to remove the reasoning block. I’d written both lines without noticing they contradict each other. If skip_special_tokens skips special tokens, why does a regex have to clean up after it?

Because <think> is registered special: false. Which raised the actual question: what does that flag change? I had a vague sense it was about “importance”, and vague senses are how you end up with two functions doing the same job badly.

So I opened GLM 5.2’s tokenizer and measured it. Everything below is output from tokenizers 0.22.2 against the real tokenizer.json, not recollection.

TL;DR

• On the encode side, special changes nothing. Both kinds are atomic, both get extracted from surrounding text, both match text a user typed.
• On the decode side it changes one thing: whether skip_special_tokens=True erases the token.
• skip_special_tokens never touches ordinary tokens, so it deletes markers and keeps whatever sat between them.
• decode(skip_special_tokens=True) is lossy. It is a rendering, not a representation.

Encoding: the flag does nothing

Four things I checked, expecting at least one difference.

Atomicity. I’d assumed special: true was what stops the tokenizer shredding <|assistant|> into <, |, assistant, |, >.

1
2

<|assistant|>    ids=[154828]  → atomic
<think>          ids=[154841]  → atomic

Both single tokens. Atomicity comes from being listed in added_tokens at all — those strings are matched whole, before BPE runs. special is a separate field on the same entry.

Extraction from surrounding text. Identical:

1
2

'before<|assistant|>after'  → ['before', '<|assistant|>', 'after']
'before<think>after'        → ['before', '<think>', 'after']

Matching text a user typed. Also identical, and this is the one worth staring at:

1
2

'please explain <|assistant|> to me'  → ['please', 'Ġexplain', 'Ġ', '<|assistant|>', 'Ġto', 'Ġme']
'please explain <think> to me'        → ['please', 'Ġexplain', 'Ġ', '<think>', 'Ġto', 'Ġme']

A user who types <|assistant|> into a chat box produces token 154828 — the genuine turn marker, indistinguishable from one your template wrote. special provides no protection here, for either class.

add_special_tokens=True vs False. No difference at all:

1
2

'hello world'    True=[14978, 1879]  False=[14978, 1879]
'<|assistant|>'  True=[154828]       False=[154828]

That surprised me until I looked at the pipeline. GLM 5.2’s post_processor is ByteLevel, not TemplateProcessing — there is no template telling it to wrap input in BOS/EOS, so add_special_tokens has nothing to add. Every role marker in a GLM prompt is there because the chat template wrote it as text.

Worth noting because it’s the opposite of the advice I gave in the chat template post, where calling the tokenizer after templating gives you two BOS. On a Llama-style tokenizer that’s real. On this one the flag is inert.

Decoding: the flag is the whole game

Take one realistic sequence — envelope written by the template, markup written by the model, ordinary prose in between:

1

[gMASK]<sop><|user|>\nwhat is 2+2<|assistant|><think>user wants arithmetic</think>It is 4.<|endoftext|>

22 tokens. Decoded both ways:

1
2
3
4
5

skip_special_tokens=False
'[gMASK]<sop><|user|>\nwhat is 2+2<|assistant|><think>user wants arithmetic</think>It is 4.<|endoftext|>'

skip_special_tokens=True
'\nwhat is 2+2<think>user wants arithmetic</think>It is 4.'

The envelope is gone. The model’s own markup is still there. And the \n survived — it was never special, just a newline the template happened to emit.

Token by token:

So the full matrix is lopsided:

Three of the four cells are the same. The flag exists for one cell.

What it doesn’t do

skip_special_tokens filters ids that are in the special set. That’s the whole mechanism, and it has no opinion about anything else:

1
2

decode([<|assistant|>, <think>, "SECRET REASONING", </think>, "answer"], skip_special_tokens=True)
'<think>SECRET REASONING</think>answer'

The reasoning text is ordinary tokens. It was never a candidate for removal.

Which is exactly why marking think tags special would be a disaster rather than a convenience. Delete the two delimiters and the reasoning prose stays, welded to the front of the answer with no boundary left to split on. The information needed to separate them would be gone by the time you saw the string.

Non-special is the only version where the model’s own structure survives to be parsed. GLM’s chat template relies on precisely that:

1

{%- set reasoning_content = content.split('</think>')[0].split('<think>')[-1] %}

String operations on decoded text. That only works because the tags are still in it.

Decoding with skip=True is lossy

The consequence I hadn’t thought about. Round-trip a sequence through decode and back:

1
2

skip=False   9 tokens → decode → encode → 9 tokens   lossless
skip=True    9 tokens → decode → encode → 6 tokens   3 lost

skip_special_tokens=False round-trips exactly. True does not, and can’t — you asked it to throw information away.

So a decoded string is a rendering for a human, not a representation you can feed back in. If you log with skip=True and later replay those logs as training data or eval inputs, you’re replaying prompts with the role markers stripped out. Nothing errors. The model just gets a differently-shaped input than it was trained on, which is the same failure mode as getting the chat template prefix wrong.

Why GLM splits its tokens where it does

Once the flag is understood as “erase this when rendering for a human” rather than “this token is important”, the design falls out.

GLM 5.2 registers 36 added tokens, ids 154820–154855. Eighteen are special, eighteen are not, and they don’t interleave — the flag flips once, at 154838, and never flips back.

Special (154820–154837) — the conversation envelope:

1
2
3
4

<|endoftext|> [MASK] [gMASK] [sMASK] <sop> <eop>
<|system|> <|user|> <|assistant|> <|observation|>
<|begin_of_image|> <|end_of_image|> <|begin_of_video|> <|end_of_video|>
<|begin_of_audio|> <|end_of_audio|> <|begin_of_transcription|> <|end_of_transcription|>

Not special (154838–154855) — markup the model writes:

1
2
3
4
5

<|code_prefix|> <|code_middle|> <|code_suffix|>
<think> </think>
<tool_call> </tool_call> <tool_response> </tool_response>
<arg_key> </arg_key> <arg_value> </arg_value>
/nothink <|begin_of_box|> <|end_of_box|> <|image|> <|video|>

The line isn’t importance. It’s who wrote the token. Your code writes the first group, by rendering a template; hiding it from a reader loses nothing. The model writes the second group, and hiding it destroys the structure of its own output.

That group is also, exactly, the extra_special_tokens list in tokenizer_config.json — I compared the two sets and they match.

Two smaller confirmations of the same logic. generation_config.json stops on [154820, 154827, 154829] — <|endoftext|>, <|user|>, <|observation|>, all three from the special half. </think> deliberately isn’t a stop token, because generation runs straight through it into the answer. And with thinking enabled the generation prompt ends on an unclosed <think>: the model wakes up already inside the block, and </think> is how it says it’s finished.

Checking any tokenizer

1
2
3
4
5
6
7
8

from tokenizers import Tokenizer
tok = Tokenizer.from_file("tokenizer.json")

for t in ["<think>", "</think>", "<|assistant|>", "<|endoftext|>"]:
    ids = tok.encode(t, add_special_tokens=False).ids
    print(f"{t:16} ids={ids}  "
          f"atomic={len(ids)==1}  "
          f"skip=True -> {tok.decode(ids, skip_special_tokens=True)!r}")

An empty last column means the token is special. A non-empty one means your code has to handle it, because nothing else will.

What I took from it

special reads like a type declaration and behaves like a display flag. It answers one question — should a human see this? — at one moment, and is inert everywhere else.

I’d been treating it as a statement about what a token is. It’s a statement about what to do with it at the very end.

• ← Previous Post
• Next Post →