fix(tokenizer): configure Rust server special tokens at load time

[jthomson04]

Summary

• upgrade the Rust server to dynamo-tokenizers 1.5.1
• construct the encode-pool tokenizer once with add_special_tokens=true, which selects the plain HuggingFace backend without a CachedTokenizer wrapper
• keep the detokenizer on default tokenizer options and leave CachedTokenizer unchanged
• add offline SGLang coverage for HF backend selection, repeated scalar parity, batch parity, and exactly-once BOS/EOS around multiple chat-boundary tokens

Test Plan

• cargo test tokenizer::tests::add_special_tokens_encoder_matches_direct_hf --lib
• cargo test --lib
• cargo test --doc
• cargo fmt --all -- --check
• cargo clippy --all-targets --all-features -- -D warnings

---

CI States

Latest PR Test (Base): ❌ Run #29266332621
Latest PR Test (Extra): ❌ Run #29266333260

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!