Summary
graphify extract aborts the entire semantic pass (exit 1, no graph.json written) when any doc in the corpus contains a literal tiktoken special-token string such as <|endoftext|>. Since label and export callflow-html both require graph.json, the whole run is lost — even though AST extraction on all code files succeeded.
Error
[graphify extract] semantic extraction failed: Encountered text corresponding to disallowed special token '<|endoftext|>'.
If you want this text to be encoded as normal text, disable the check for this token by passing `disallowed_special=()`.
[graphify extract] error: all semantic chunks failed for backend 'ollama' (N uncached files)
Root cause
graphify/llm.py calls _TOKENIZER.encode(content) for token estimation without passing disallowed_special=():
llm.py:1455 — return len(_TOKENIZER.encode(content)) + ...llm.py:1474 — return len(_TOKENIZER.encode(content)) + ...
tiktoken's default encode() raises ValueError on any of the model's special-token strings appearing as ordinary text. Documentation about LLMs/tokenizers routinely contains these literally (HuggingFace tokenizer docs, unsloth references, prompt-template examples, this issue itself). One such file anywhere in the corpus kills the whole batch.
Impact
Common on real repos: any repo whose docs discuss tokenizers/LLMs. In my case a monorepo with ~1,600 doc files had ~20 containing <|endoftext|> (all in tokenizer skill docs) — the entire extract failed until those files were fenced out via .graphifyignore.
Fix
Pass disallowed_special=() at both call sites so special-token strings are counted as ordinary text (correct for a token-estimate):
len(_TOKENIZER.encode(content, disallowed_special=()))
This is purely a token-count estimate, so treating the sentinel as normal text is semantically correct and can only make the estimate more accurate (it currently can't produce a number at all).
Repro
mkdir /tmp/gfy-repro && printf 'The GPT end-of-text token is <|endoftext|> in the vocab.\n' > /tmp/gfy-repro/doc.md
printf 'def f():\n return 1\n' > /tmp/gfy-repro/code.py
graphify extract /tmp/gfy-repro --backend ollama --model <any>
# -> "Encountered text corresponding to disallowed special token '<|endoftext|>'" ; no graph.json
Version: graphifyy 0.9.5
Summary
graphify extractaborts the entire semantic pass (exit 1, nograph.jsonwritten) when any doc in the corpus contains a literal tiktoken special-token string such as<|endoftext|>. Sincelabelandexport callflow-htmlboth requiregraph.json, the whole run is lost — even though AST extraction on all code files succeeded.Error
Root cause
graphify/llm.pycalls_TOKENIZER.encode(content)for token estimation without passingdisallowed_special=():llm.py:1455—return len(_TOKENIZER.encode(content)) + ...llm.py:1474—return len(_TOKENIZER.encode(content)) + ...tiktoken's default
encode()raisesValueErroron any of the model's special-token strings appearing as ordinary text. Documentation about LLMs/tokenizers routinely contains these literally (HuggingFace tokenizer docs, unsloth references, prompt-template examples, this issue itself). One such file anywhere in the corpus kills the whole batch.Impact
Common on real repos: any repo whose docs discuss tokenizers/LLMs. In my case a monorepo with ~1,600 doc files had ~20 containing
<|endoftext|>(all in tokenizer skill docs) — the entire extract failed until those files were fenced out via.graphifyignore.Fix
Pass
disallowed_special=()at both call sites so special-token strings are counted as ordinary text (correct for a token-estimate):This is purely a token-count estimate, so treating the sentinel as normal text is semantically correct and can only make the estimate more accurate (it currently can't produce a number at all).
Repro
Version: graphifyy 0.9.5