Tokenizer plugins

[wjones127]

It won't be sustainable to statically compile tokenizers into Lance. It also would be annoying to require them all to have feature flags in Lance.

We currently use Tantivy for tokenizer, so it might make sense for us to re-use their API. There is some discussion on their side for using a plugin API. It might make sense for us to help with that and re-use that same mechanism. That way tokenizer providers don't have to do anything extra to support Lance if they are supporting tantivy.

Tantivy has a stable API for tokenizeres here: https://github.com/quickwit-oss/tantivy/blob/main/tokenizer-api/src/lib.rs

(It's a Rust API though. We might want to create a stable C API, which will be more cross compatible between compilers.)

quickwit-oss/tantivy-py#358
quickwit-oss/tantivy-py#200

[chenkovsky]

@wjones127 I created a C API draft. https://gist.github.com/chenkovsky/40a5e6078fe7857ba3a2c1b2c624b4b6 . if you think it's feasible. I will create a PR. I have done a PoC on this API.

[wjones127]

During our sync meeting on January 15, I proposed that we:

1. Vote on the format change to allow tokenizer plugins (protobuf changes)
2. Implement tokenizer plugins for Lindera and Jieba
3. Benchmark whether there are any performance regressions
4. Once on par, deprecate and eventually remove feature flags for Lindera and Jieba.